Started by user Ritesh Agarwal [EnvInject] - Loading node environment variables. Building remotely on magma-sc1403-32g-24c (magma_slave) in workspace /data/workspace/magma_temp_job3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins6660245840433403731.sh ++ echo magma + desc=magma + echo Desc: 7.1.0-1913 - magma Desc: 7.1.0-1913 - magma Success build forhudson.tasks.Shell@654f4e00 [description-setter] Description set: 7.1.0-1913 - magma Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@120bb0d0 Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/magma_temp_job3 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 5c9788c0894619c5f6243129f316acef347cfb36 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 5c9788c0894619c5f6243129f316acef347cfb36 > /usr/bin/git rev-list 938c3deacda0a8c3bdd733cbfe4cc7f88edb3b4f # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #259 jenkins-magma_temp_job3-259 # timeout=10 No emails were triggered. [magma_temp_job3] $ /bin/bash /tmp/jenkins2227165168212364404.sh [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 data_path:/data [servers] 1:172.23.110.64 2:172.23.110.65 3:172.23.110.66 4:172.23.110.67 5:172.23.110.68 [membase] rest_username:Administrator rest_password:password [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins780759696478840926.sh + echo '' + echo '##############################################' ############################################## + echo 'Installing dependencies from requirements.txt' Installing dependencies from requirements.txt + echo '##############################################' ############################################## + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + echo '#################### Done ####################' #################### Done #################### + echo '' + echo version=7.1.0-1913 version=7.1.0-1913 + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... + cd testrunner + git checkout master Already on 'master' + git pull origin master From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. + py_executable=python3 + python3 scripts/new_install.py -i /tmp/magma_temp_job3.ini -p timeout=1200,get-cbcollect-info=True,version=7.1.0-1913,product=cb,debug_logs=False,url=,install_debug_info=true 2021-12-14 19:47:56,534 - root - WARNING - URL: is not valid, will use version to locate build 2021-12-14 19:47:56,536 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-14 19:47:56,681 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-14 19:47:57,028 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:47:57,408 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:47:57,410 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-14 19:47:57,566 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-14 19:47:57,965 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:47:58,345 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:47:58,347 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-14 19:47:58,503 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-14 19:47:58,925 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:47:59,307 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:47:59,309 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-14 19:47:59,464 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-14 19:47:59,846 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:00,221 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:00,222 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-14 19:48:00,379 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-14 19:48:00,772 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:01,158 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:01,160 - root - INFO - SSH Connecting to 172.23.110.64 with username:root, attempt#1 of 5 2021-12-14 19:48:01,307 - root - INFO - SSH Connected to 172.23.110.64 as root 2021-12-14 19:48:01,647 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:02,029 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:02,031 - root - INFO - SSH Connecting to 172.23.110.65 with username:root, attempt#1 of 5 2021-12-14 19:48:02,181 - root - INFO - SSH Connected to 172.23.110.65 as root 2021-12-14 19:48:02,559 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:02,949 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:02,950 - root - INFO - SSH Connecting to 172.23.110.66 with username:root, attempt#1 of 5 2021-12-14 19:48:03,104 - root - INFO - SSH Connected to 172.23.110.66 as root 2021-12-14 19:48:03,497 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:03,878 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:03,880 - root - INFO - SSH Connecting to 172.23.110.67 with username:root, attempt#1 of 5 2021-12-14 19:48:04,033 - root - INFO - SSH Connected to 172.23.110.67 as root 2021-12-14 19:48:04,399 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:04,789 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:04,790 - root - INFO - SSH Connecting to 172.23.110.68 with username:root, attempt#1 of 5 2021-12-14 19:48:04,936 - root - INFO - SSH Connected to 172.23.110.68 as root 2021-12-14 19:48:05,305 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:48:05,691 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-12-14 19:48:05,692 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,695 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,695 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,762 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,762 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,763 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,763 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,765 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,765 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,766 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,766 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,767 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,767 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,768 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,768 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,769 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,770 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,771 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:05,771 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:48:05,772 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/neo/1913/couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm is live 2021-12-14 19:48:13,772 - root - INFO - Done with uninstall on 172.23.110.68. 2021-12-14 19:49:20,767 - root - INFO - Done with uninstall on 172.23.110.67. 2021-12-14 19:49:22,232 - root - INFO - Done with uninstall on 172.23.110.65. 2021-12-14 19:49:23,315 - root - INFO - Done with uninstall on 172.23.110.66. 2021-12-14 19:49:42,344 - root - INFO - Done with uninstall on 172.23.110.64. 2021-12-14 19:49:46,430 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:46,458 - root - INFO - command executed successfully with root 2021-12-14 19:49:48,669 - root - INFO - running command.raw on 172.23.110.64: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:48,697 - root - INFO - command executed successfully with root 2021-12-14 19:49:49,341 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:49,366 - root - INFO - command executed successfully with root 2021-12-14 19:49:50,115 - root - INFO - running command.raw on 172.23.110.65: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:50,139 - root - INFO - command executed successfully with root 2021-12-14 19:49:50,771 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:50,796 - root - INFO - command executed successfully with root 2021-12-14 19:49:51,470 - root - INFO - running command.raw on 172.23.110.66: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:51,494 - root - INFO - command executed successfully with root 2021-12-14 19:49:52,136 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:52,162 - root - INFO - command executed successfully with root 2021-12-14 19:49:52,915 - root - INFO - running command.raw on 172.23.110.67: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:52,939 - root - INFO - command executed successfully with root 2021-12-14 19:49:53,586 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:53,613 - root - INFO - command executed successfully with root 2021-12-14 19:49:54,337 - root - INFO - running command.raw on 172.23.110.68: cd /tmp/ && wc -c couchbase-server-enterprise-debuginfo-7.1.0-1913-centos7.x86_64.rpm 2021-12-14 19:49:54,362 - root - INFO - command executed successfully with root 2021-12-14 19:51:25,505 - root - INFO - Done with install on 172.23.110.67. 2021-12-14 19:51:25,505 - root - INFO - Waiting for couchbase to be reachable 2021-12-14 19:51:25,515 - root - INFO - running command.raw on 172.23.110.67: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.67 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-14 19:51:25,935 - root - INFO - Done with install on 172.23.110.65. 2021-12-14 19:51:25,935 - root - INFO - Waiting for couchbase to be reachable 2021-12-14 19:51:25,945 - root - INFO - running command.raw on 172.23.110.65: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.65 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-14 19:51:25,968 - root - INFO - command executed successfully with root 2021-12-14 19:51:25,973 - root - INFO - running command.raw on 172.23.110.67: rm -rf /data/* 2021-12-14 19:51:26,414 - root - INFO - command executed successfully with root 2021-12-14 19:51:26,418 - root - INFO - running command.raw on 172.23.110.65: rm -rf /data/* 2021-12-14 19:51:27,502 - root - INFO - Done with install on 172.23.110.66. 2021-12-14 19:51:27,503 - root - INFO - Waiting for couchbase to be reachable 2021-12-14 19:51:27,512 - root - INFO - running command.raw on 172.23.110.66: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.66 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-14 19:51:27,598 - root - INFO - Done with install on 172.23.110.68. 2021-12-14 19:51:27,598 - root - INFO - Waiting for couchbase to be reachable 2021-12-14 19:51:27,607 - root - INFO - running command.raw on 172.23.110.68: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.68 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-14 19:51:28,016 - root - INFO - command executed successfully with root 2021-12-14 19:51:28,021 - root - INFO - running command.raw on 172.23.110.66: rm -rf /data/* 2021-12-14 19:51:28,091 - root - INFO - command executed successfully with root 2021-12-14 19:51:28,095 - root - INFO - running command.raw on 172.23.110.68: rm -rf /data/* 2021-12-14 19:51:28,119 - root - INFO - command executed successfully with root 2021-12-14 19:51:28,119 - root - INFO - running command.raw on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-14 19:51:28,182 - root - INFO - command executed successfully with root 2021-12-14 19:51:28,182 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-14 19:51:28,494 - root - INFO - Done with install on 172.23.110.64. 2021-12-14 19:51:28,494 - root - INFO - Waiting for couchbase to be reachable 2021-12-14 19:51:28,503 - root - INFO - running command.raw on 172.23.110.64: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.110.64 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-12-14 19:51:28,972 - root - INFO - command executed successfully with root 2021-12-14 19:51:28,976 - root - INFO - running command.raw on 172.23.110.64: rm -rf /data/* 2021-12-14 19:51:30,839 - root - INFO - Setting data_path: /data: status True 2021-12-14 19:51:31,849 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.68 2021-12-14 19:51:31,849 - root - INFO - pools/default params : memoryQuota=172436 2021-12-14 19:51:31,854 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-14 19:51:31,854 - root - INFO - /node/controller/setupServices params on 172.23.110.68: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-14 19:51:31,887 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-14 19:51:31,887 - root - INFO - settings/web params on 172.23.110.68:8091:port=8091&username=Administrator&password=password 2021-12-14 19:51:31,978 - root - INFO - --> status:True 2021-12-14 19:51:31,978 - root - INFO - Done with init on 172.23.110.68. 2021-12-14 19:51:32,243 - root - INFO - Done with cleanup on 172.23.110.68. 2021-12-14 19:51:55,725 - root - INFO - command executed successfully with root 2021-12-14 19:51:55,726 - root - INFO - running command.raw on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-14 19:51:55,750 - root - INFO - command executed successfully with root 2021-12-14 19:51:55,751 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-14 19:51:58,577 - root - INFO - Setting data_path: /data: status True 2021-12-14 19:51:59,583 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.67 2021-12-14 19:51:59,583 - root - INFO - pools/default params : memoryQuota=172436 2021-12-14 19:51:59,588 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-14 19:51:59,588 - root - INFO - /node/controller/setupServices params on 172.23.110.67: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-14 19:51:59,622 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-14 19:51:59,622 - root - INFO - settings/web params on 172.23.110.67:8091:port=8091&username=Administrator&password=password 2021-12-14 19:51:59,726 - root - INFO - --> status:True 2021-12-14 19:51:59,727 - root - INFO - Done with init on 172.23.110.67. 2021-12-14 19:52:00,040 - root - INFO - Done with cleanup on 172.23.110.67. 2021-12-14 19:52:36,626 - root - INFO - command executed successfully with root 2021-12-14 19:52:36,626 - root - INFO - running command.raw on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-14 19:52:36,692 - root - INFO - command executed successfully with root 2021-12-14 19:52:36,694 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-14 19:52:39,777 - root - INFO - Setting data_path: /data: status True 2021-12-14 19:52:40,784 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.66 2021-12-14 19:52:40,784 - root - INFO - pools/default params : memoryQuota=172436 2021-12-14 19:52:40,789 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-14 19:52:40,789 - root - INFO - /node/controller/setupServices params on 172.23.110.66: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-14 19:52:40,820 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-14 19:52:40,821 - root - INFO - settings/web params on 172.23.110.66:8091:port=8091&username=Administrator&password=password 2021-12-14 19:52:40,924 - root - INFO - --> status:True 2021-12-14 19:52:40,924 - root - INFO - Done with init on 172.23.110.66. 2021-12-14 19:52:40,971 - root - INFO - Done with cleanup on 172.23.110.66. 2021-12-14 19:52:41,227 - root - INFO - command executed successfully with root 2021-12-14 19:52:41,227 - root - INFO - running command.raw on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-14 19:52:41,254 - root - INFO - command executed successfully with root 2021-12-14 19:52:41,254 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-14 19:52:43,947 - root - INFO - Setting data_path: /data: status True 2021-12-14 19:52:44,952 - root - INFO - Setting KV memory quota as 172436 MB on 172.23.110.65 2021-12-14 19:52:44,952 - root - INFO - pools/default params : memoryQuota=172436 2021-12-14 19:52:44,957 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-14 19:52:44,958 - root - INFO - /node/controller/setupServices params on 172.23.110.65: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-14 19:52:44,992 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-14 19:52:44,992 - root - INFO - settings/web params on 172.23.110.65:8091:port=8091&username=Administrator&password=password 2021-12-14 19:52:45,090 - root - INFO - --> status:True 2021-12-14 19:52:45,090 - root - INFO - Done with init on 172.23.110.65. 2021-12-14 19:52:45,143 - root - INFO - Done with cleanup on 172.23.110.65. 2021-12-14 19:52:56,402 - root - INFO - command executed successfully with root 2021-12-14 19:52:56,402 - root - INFO - running command.raw on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-14 19:52:56,472 - root - INFO - command executed successfully with root 2021-12-14 19:52:56,472 - root - INFO - /nodes/self/controller/settings : path=%2Fdata 2021-12-14 19:52:59,270 - root - INFO - Setting data_path: /data: status True 2021-12-14 19:53:00,308 - root - INFO - Setting KV memory quota as 137949 MB on 172.23.110.64 2021-12-14 19:53:00,308 - root - INFO - pools/default params : memoryQuota=137949 2021-12-14 19:53:00,314 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-12-14 19:53:00,315 - root - INFO - /node/controller/setupServices params on 172.23.110.64: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-12-14 19:53:00,347 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-12-14 19:53:00,347 - root - INFO - settings/web params on 172.23.110.64:8091:port=8091&username=Administrator&password=password 2021-12-14 19:53:00,458 - root - INFO - --> status:True 2021-12-14 19:53:00,458 - root - INFO - Done with init on 172.23.110.64. 2021-12-14 19:53:00,490 - root - INFO - Done with cleanup on 172.23.110.64. 2021-12-14 19:53:04,619 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-14 19:53:04,647 - root - INFO - cluster:C1 node:172.23.110.64:8091 version:7.1.0-1913-enterprise aFamily:inet services:['kv'] 2021-12-14 19:53:04,647 - root - INFO - cluster:C2 node:172.23.110.65:8091 version:7.1.0-1913-enterprise aFamily:inet services:['kv'] 2021-12-14 19:53:04,647 - root - INFO - cluster:C3 node:172.23.110.66:8091 version:7.1.0-1913-enterprise aFamily:inet services:['kv'] 2021-12-14 19:53:04,647 - root - INFO - cluster:C4 node:172.23.110.67:8091 version:7.1.0-1913-enterprise aFamily:inet services:['kv'] 2021-12-14 19:53:04,647 - root - INFO - cluster:C5 node:172.23.110.68:8091 version:7.1.0-1913-enterprise aFamily:inet services:['kv'] 2021-12-14 19:53:04,647 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-14 19:53:04,647 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-14 19:53:04,647 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-14 19:53:04,647 - root - INFO - INSTALL COMPLETED ON: 172.23.110.64 2021-12-14 19:53:04,647 - root - INFO - INSTALL COMPLETED ON: 172.23.110.65 2021-12-14 19:53:04,647 - root - INFO - INSTALL COMPLETED ON: 172.23.110.66 2021-12-14 19:53:04,648 - root - INFO - INSTALL COMPLETED ON: 172.23.110.67 2021-12-14 19:53:04,648 - root - INFO - INSTALL COMPLETED ON: 172.23.110.68 2021-12-14 19:53:04,648 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-12-14 19:53:04,648 - root - INFO - TOTAL INSTALL TIME = 308 seconds + status=0 + cd ../.. + rm -rf tr_for_install + git clone https://github.com/gradle/guides.git Cloning into 'guides'... [magma_temp_job3] $ /bin/sh -xe /tmp/jenkins5881465293564226193.sh + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest' Starting a Gradle Daemon, 43 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.2.1 Using Java_client :: 3.2.2 Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/ssh.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar scripts/eagles_all_around.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main scripts/install.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/magma_temp_job3/build/classes/java/main:/data/workspace/magma_temp_job3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-blob/12.14.0/75d45d21dc208fa369abc832354470464c859d67/azure-storage-blob-12.14.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.2.1/170f0e036a91f8823da2f9c46f5fa4a6c8e012b4/couchbase-transactions-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.2.2/f5e9f5f4f4791edbb5680d4ec7812673b0c79a2b/java-client-3.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-internal-avro/12.1.0/c5882c519f8ee15c528e9d0396ec7535da7d4ecd/azure-storage-internal-avro-12.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-storage-common/12.13.0/ef8910849375def2b715ea64429136e694fb7ad6/azure-storage-common-12.13.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core-http-netty/1.11.0/2177ae2dc9cff849e49f9180bf6b95b8e2c78e1b/azure-core-http-netty-1.11.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.azure/azure-core/1.20.0/a98c6bd18aa2066ecd8b39bf7ac51bd8e7307851/azure-core-1.20.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.32/cdcff33940d9f2de763bc41ea05a0be5941176c3/slf4j-api-1.7.32.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.2.2/74e42697de31022743d5983542d2b22efe6830f0/core-io-2.2.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.dataformat/jackson-dataformat-xml/2.12.4/15c743856696c0239f2c51d8d19d9f97f034713/jackson-dataformat-xml-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.module/jackson-module-jaxb-annotations/2.12.4/5e43703aae1a9843dfd7df0a0ad6cbfedcaff67f/jackson-module-jaxb-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.12.4/6a1bd259b6c4e3f9219ec8ec0be55ed11eed0c/jackson-core-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.12.4/b1174c05d4ded121a7eaeed3f148709f9585b981/jackson-datatype-jsr310-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.12.4/69206e02e6a696034f06a59d3ddbfbba5a4cd81/jackson-databind-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.12.4/752cf9a2562ac2c012e48057e3a4c17dad66c66e/jackson-annotations-2.12.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty/1.0.10/c54f6c92628f396076f74945424319746bbb37f8/reactor-netty-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http-brave/1.0.10/30aebc029ee25f5e0c2d1c0a798b17e926f61b8/reactor-netty-http-brave-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-http/1.0.10/f34196d8778243d2ac4a4ae7cc74f110d1073803/reactor-netty-http-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor.netty/reactor-netty-core/1.0.10/dcb5d53dc07a6f660060f0b55c40a1e9944a6fe1/reactor-netty-core-1.0.10.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.9/820332aa7b0fe3a8dfe14f58fc16e49ad178291/reactor-core-3.4.9.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-tcnative-boringssl-static/2.0.40.Final/6b73a163c13ed76921892d28eb81235f4b41e40a/netty-tcnative-boringssl-static-2.0.40.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler-proxy/4.1.67.Final/2528292c49bd15f1b328dda25a7a75744d6f0991/netty-handler-proxy-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http2/4.1.67.Final/e2a0c27f396035fab7d932031d6f337244369495/netty-codec-http2-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-http/4.1.67.Final/e282137917c67332fa9a414df89f89a93487aede/netty-codec-http-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns-native-macos/4.1.66.Final/8ca314d87b202a4dd94d810d2385b22555dd801c/netty-resolver-dns-native-macos-4.1.66.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver-dns/4.1.66.Final/7b74815fb1403e5747c872c6eee2a07e7a700d30/netty-resolver-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-handler/4.1.67.Final/62640a6524e1c08d6e8ac06556892b5e1362392f/netty-handler-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-epoll/4.1.67.Final/ff955604e1edb8b13dc855ccbc84acb5fc1d989/netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-kqueue/4.1.67.Final/7a88ccb4237d1ff091be544860ae729f5b568ac/netty-transport-native-kqueue-4.1.67.Final-osx-x86_64.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport-native-unix-common/4.1.67.Final/c6cca7ba897e1ac65e0742654ed10064401dbd2f/netty-transport-native-unix-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-socks/4.1.67.Final/a4fe0487451dc262daaa024224fa207d088b8687/netty-codec-socks-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec-dns/4.1.66.Final/390ff96a0e1f7c626cb52c119a1a1dfd0784d193/netty-codec-dns-4.1.66.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-codec/4.1.67.Final/292e818822a4fca4e9f3ad711cdd46362f69590a/netty-codec-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-transport/4.1.67.Final/e8b502618f960cb1f8cdb7dca92d0878576ad7c6/netty-transport-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-buffer/4.1.67.Final/4b6217b05792fe6e2bd080c054138b2c20ae1b37/netty-buffer-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.fasterxml.woodstox/woodstox-core/6.2.4/16b9f8ab972e67eb21872ea2c40046249d543989/woodstox-core-6.2.4.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.woodstox/stax2-api/4.2.1/a3f7325c52240418c2ba257b103c3c550e140c83/stax2-api-4.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-resolver/4.1.67.Final/bb9041d5f85f9a6270f5378d0becd243de8cfeac/netty-resolver-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/io.netty/netty-common/4.1.67.Final/a86a19033588a07e159ff95818845b1ec86b2281/netty-common-4.1.67.Final.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.xml.bind/jakarta.xml.bind-api/2.3.2/8d49996a4338670764d7ca4b85a1c4ccf7fe665d/jakarta.xml.bind-api-2.3.2.jar:/root/.gradle/caches/modules-2/files-2.1/jakarta.activation/jakarta.activation-api/1.2.1/562a587face36ec7eff2db7f2fc95425c6602bc1/jakarta.activation-api-1.2.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave-instrumentation-http/5.13.3/fe70809f06c786b171e4597747c6f5a8c911fe5f/brave-instrumentation-http-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.brave/brave/5.13.3/2d8ecb2352108f95b3e66ccc8027e9343ad9852c/brave-5.13.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter-brave/2.16.3/4d5017d71e4de139b6a31612cfd837b7c71d288c/zipkin-reporter-brave-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.reporter2/zipkin-reporter/2.16.3/7e43d8be3376d305c355d969e8b9f3a62221380/zipkin-reporter-2.16.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.zipkin.zipkin2/zipkin/2.23.2/1c2c7f2e91a3749311f7f75d0535d14ba2e2f6/zipkin-2.23.2.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/magma_temp_job3.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey -m rest > Task :compileJava warning: unknown enum constant javax.annotation.meta.When.MAYBE reason: class file for javax.annotation.meta.When not found Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. 1 warning > Task :testrunner Global Test input params: {'bucket_eviction_policy': 'fullEviction', 'bucket_storage': 'magma', 'cluster_name': 'magma_temp_job3', 'ini': '/tmp/magma_temp_job3.ini', 'num_nodes': 5, 'rerun': 'False'} Logs will be stored at /data/workspace/magma_temp_job3/logs/testrunner-21-Dec-14_19-53-30/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job3.ini bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=3,graceful=True,skip_cleanup=True,num_items=40000000,num_buckets=1,bucket_names=GleamBook,doc_size=1536,bucket_type=membase,eviction_policy=fullEviction,iterations=1,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=100,pc=10,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=200000,ramQuota=102400,doc_ops=create:update:delete:read,mutation_perc=100,rebl_ops_rate=50000,key_type=RandomKey' Test Input params: {'fts_nodes': '0', 'doc_size': '1536', 'ops_rate': '200000', 'key_type': 'RandomKey', 'bucket_type': 'membase', 'num_collections': '50', 'maxttl': '10', 'graceful': 'True', 'rerun': 'False', 'num_nodes': 5, 'doc_ops': 'create:update:delete:read', 'ramQuota': '102400', 'eviction_policy': 'fullEviction', 'sdk_timeout': '60', 'case_number': 1, 'num_indexes': '100', 'cluster_name': 'magma_temp_job3', 'batch_size': '1000', 'ini': '/tmp/magma_temp_job3.ini', 'num_buckets': '1', 'log_level': 'debug', 'bucket_storage': 'magma', 'rebl_ops_rate': '50000', 'bucket_eviction_policy': 'fullEviction', 'iterations': '1', 'logs_folder': '/data/workspace/magma_temp_job3/logs/testrunner-21-Dec-14_19-53-30/test_1', 'skip_cleanup': 'True', 'nodes_init': '3', 'pc': '10', 'num_items': '40000000', 'cbas_nodes': '0', 'assert_crashes_on_load': 'True', 'mutation_perc': '100', 'bucket_names': 'GleamBook', 'randomize_doc_size': 'False', 'index_nodes': '0', 'randomize_value': 'True', 'infra_log_level': 'debug', 'key_size': '18'} SystemTestMagma (aGoodDoctor.Hospital.Murphy) ... 2021-12-14 19:53:37,278 | infra | INFO | MainThread | [task_manager:__init__:13] Initiating TaskManager with 30 threads 2021-12-14 19:53:37,282 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-12-14 19:53:37,282 | test | INFO | MainThread | [basetestcase:log_setup_status:690] ========= Murphy setup started for test #1 SystemTestMagma ========= 2021-12-14 19:53:37,522 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:53:37,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:37,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:53:38,009 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:38,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:53:38,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:53:38,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:53:38,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:53:38,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:53:38,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:53:38,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:53:38,200 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 19:53:38,201 | test | INFO | MainThread | [basetestcase:log_setup_status:690] ========= BaseTestCase setup started for test #1 SystemTestMagma ========= 2021-12-14 19:53:38,203 | test | DEBUG | MainThread | [basetestcase:setUp:332] Cleaning up cluster 2021-12-14 19:53:38,592 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 120secs 2021-12-14 19:53:38,677 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-14 19:53:38,677 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:53:38,696 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-14 19:53:38,698 | test | WARNING | MainThread | [basetestcase:tearDownEverything:615] CLEANUP WAS SKIPPED 2021-12-14 19:53:38,700 | infra | INFO | MainThread | [basetestcase:tearDownEverything:658] ========== tasks in thread pool ========== 2021-12-14 19:53:38,700 | infra | INFO | MainThread | [basetestcase:tearDownEverything:660] ========================================== 2021-12-14 19:53:38,700 | test | INFO | MainThread | [basetestcase:initialize_cluster:440] Initializing cluster : C1 2021-12-14 19:53:38,704 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:53:38,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:39,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:53:39,171 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:39,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:53:39,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:53:39,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:53:39,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:53:39,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:53:39,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:53:39,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:53:39,372 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:53:39,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:39,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:53:39,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:39,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:53:39,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:53:39,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:53:39,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:53:39,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:53:39,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:53:40,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:53:40,046 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-14 19:53:40,078 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:53:40,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-14 19:53:40,163 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.64 - Running systemd command 2021-12-14 19:53:40,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl stop couchbase-server.service 2021-12-14 19:53:42,331 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-14 19:53:42,332 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 19:53:42,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-14 19:53:42,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:53:42,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-14 19:53:42,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:53:42,446 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:53:42,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:42,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:53:42,910 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:42,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:53:42,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:53:42,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:53:42,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:53:43,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:53:43,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:53:43,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:53:43,061 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.64 - Checking if couchbase-server is installed 2021-12-14 19:53:43,085 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.64 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:53:43,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /opt/couchbase/VERSION.txt 2021-12-14 19:53:43,160 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.64 - Running systemd command 2021-12-14 19:53:43,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: systemctl start couchbase-server.service 2021-12-14 19:53:43,193 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.64 port:8091 ssh_username:root 2021-12-14 19:53:43,193 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0x9 type=client open_count=1 channel=[id: 0x33578964, 0.0.0.0/0.0.0.0:53821] timeout=300.0>) 2021-12-14 19:53:43,201 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:43,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:43,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xa type=client open_count=1 channel=[id: 0xb6e04abb, 0.0.0.0/0.0.0.0:54567] timeout=300.0>) 2021-12-14 19:53:45,677 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:45,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:45,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0x27551c9b, 0.0.0.0/0.0.0.0:35585] timeout=300.0>) 2021-12-14 19:53:47,703 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:47,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:47,724 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0x82d86787, 0.0.0.0/0.0.0.0:55622] timeout=300.0>) 2021-12-14 19:53:49,730 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:49,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:49,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.64:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xc25e5f0e, 0.0.0.0/0.0.0.0:48763] timeout=300.0>) 2021-12-14 19:53:51,750 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:51,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:51,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:53:53,789 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:53:53,802 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 19:53:53,806 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 19:53:53,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:54,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 19:53:54,319 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:54,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 19:53:54,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 19:53:54,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 19:53:54,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 19:53:54,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 19:53:54,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 19:53:54,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 19:53:54,487 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 19:53:54,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:54,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 19:53:54,993 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:54,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 19:53:55,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 19:53:55,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 19:53:55,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 19:53:55,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 19:53:55,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 19:53:55,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 19:53:55,137 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-14 19:53:55,157 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:53:55,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-14 19:53:55,217 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.65 - Running systemd command 2021-12-14 19:53:55,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl stop couchbase-server.service 2021-12-14 19:53:57,773 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-14 19:53:57,773 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 19:53:57,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-14 19:53:57,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:53:57,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-14 19:53:57,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:53:57,868 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 19:53:57,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:53:58,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 19:53:58,348 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:53:58,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 19:53:58,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 19:53:58,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 19:53:58,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 19:53:58,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 19:53:58,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 19:53:58,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 19:53:58,477 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.65 - Checking if couchbase-server is installed 2021-12-14 19:53:58,500 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.65 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:53:58,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /opt/couchbase/VERSION.txt 2021-12-14 19:53:58,552 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.65 - Running systemd command 2021-12-14 19:53:58,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: systemctl start couchbase-server.service 2021-12-14 19:53:58,581 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.65 port:8091 ssh_username:root 2021-12-14 19:53:58,582 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x73cef53d, 0.0.0.0/0.0.0.0:52631] timeout=300.0>) 2021-12-14 19:53:58,588 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:53:58,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:53:58,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x5b170fe1, 0.0.0.0/0.0.0.0:54169] timeout=300.0>) 2021-12-14 19:54:00,608 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:00,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:00,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x9b429c2c, 0.0.0.0/0.0.0.0:57210] timeout=300.0>) 2021-12-14 19:54:02,625 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:02,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:02,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xd31209b0, 0.0.0.0/0.0.0.0:35225] timeout=300.0>) 2021-12-14 19:54:04,640 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:04,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:04,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.65:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xfc300390, 0.0.0.0/0.0.0.0:32876] timeout=300.0>) 2021-12-14 19:54:06,657 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.65:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:06,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:06,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:54:08,687 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:54:08,701 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 19:54:08,706 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 19:54:08,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:09,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 19:54:09,263 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:09,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 19:54:09,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 19:54:09,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 19:54:09,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 19:54:09,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 19:54:09,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 19:54:09,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 19:54:09,434 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 19:54:09,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:09,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 19:54:09,950 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:09,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 19:54:09,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 19:54:09,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 19:54:10,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 19:54:10,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 19:54:10,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 19:54:10,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 19:54:10,082 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-14 19:54:10,104 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:10,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:10,167 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.66 - Running systemd command 2021-12-14 19:54:10,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl stop couchbase-server.service 2021-12-14 19:54:12,312 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-14 19:54:12,312 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 19:54:12,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-14 19:54:12,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:12,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-14 19:54:12,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:12,408 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 19:54:12,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:12,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 19:54:12,924 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:12,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 19:54:12,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 19:54:12,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 19:54:12,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 19:54:13,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 19:54:13,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 19:54:13,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 19:54:13,075 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.66 - Checking if couchbase-server is installed 2021-12-14 19:54:13,098 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.66 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:13,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:13,163 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.66 - Running systemd command 2021-12-14 19:54:13,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: systemctl start couchbase-server.service 2021-12-14 19:54:13,194 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.66 port:8091 ssh_username:root 2021-12-14 19:54:13,194 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0xb406810a, 0.0.0.0/0.0.0.0:35745] timeout=300.0>) 2021-12-14 19:54:13,200 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:13,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:13,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x06a77a13, 0.0.0.0/0.0.0.0:48934] timeout=300.0>) 2021-12-14 19:54:15,216 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:15,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:15,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x54a90903, 0.0.0.0/0.0.0.0:33718] timeout=300.0>) 2021-12-14 19:54:17,230 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:17,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:17,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x304e3c59, 0.0.0.0/0.0.0.0:58082] timeout=300.0>) 2021-12-14 19:54:19,247 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:19,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:19,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.66:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x7e591d38, 0.0.0.0/0.0.0.0:34136] timeout=300.0>) 2021-12-14 19:54:21,266 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.66:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:21,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:21,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:54:23,302 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:54:23,315 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 19:54:23,319 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-14 19:54:23,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:23,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-14 19:54:23,835 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:23,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-14 19:54:23,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-14 19:54:23,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-14 19:54:23,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-14 19:54:23,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-14 19:54:23,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-14 19:54:23,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-14 19:54:24,022 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-14 19:54:24,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:24,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-14 19:54:24,520 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:24,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-14 19:54:24,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-14 19:54:24,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-14 19:54:24,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-14 19:54:24,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-14 19:54:24,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-14 19:54:24,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-14 19:54:24,677 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-14 19:54:24,704 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:24,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:24,779 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.67 - Running systemd command 2021-12-14 19:54:24,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl stop couchbase-server.service 2021-12-14 19:54:26,710 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-14 19:54:26,710 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-14 19:54:26,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-14 19:54:26,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:26,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-14 19:54:26,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:26,809 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-14 19:54:26,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:27,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-14 19:54:27,293 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:27,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-14 19:54:27,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-14 19:54:27,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-14 19:54:27,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-14 19:54:27,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-14 19:54:27,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-14 19:54:27,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-14 19:54:27,427 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.67 - Checking if couchbase-server is installed 2021-12-14 19:54:27,448 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.67 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:27,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:27,512 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.67 - Running systemd command 2021-12-14 19:54:27,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: systemctl start couchbase-server.service 2021-12-14 19:54:27,540 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.67 port:8091 ssh_username:root 2021-12-14 19:54:27,540 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0xf3f50af4, 0.0.0.0/0.0.0.0:36313] timeout=300.0>) 2021-12-14 19:54:27,546 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:27,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:27,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x8165ac2e, 0.0.0.0/0.0.0.0:50032] timeout=300.0>) 2021-12-14 19:54:29,562 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:29,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:29,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x7e08c253, 0.0.0.0/0.0.0.0:54645] timeout=300.0>) 2021-12-14 19:54:31,578 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:31,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:31,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.67:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0xcca97610, 0.0.0.0/0.0.0.0:36043] timeout=300.0>) 2021-12-14 19:54:33,594 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.67:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:33,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:33,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:54:35,887 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:54:35,900 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-14 19:54:35,903 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-14 19:54:36,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:36,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-14 19:54:36,391 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:36,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-14 19:54:36,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-14 19:54:36,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-14 19:54:36,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-14 19:54:36,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-14 19:54:36,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-14 19:54:36,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-14 19:54:36,579 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-14 19:54:36,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:36,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-14 19:54:37,055 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:37,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-14 19:54:37,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-14 19:54:37,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-14 19:54:37,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-14 19:54:37,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-14 19:54:37,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-14 19:54:37,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-14 19:54:37,181 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-14 19:54:37,203 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:37,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:37,269 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3806] 172.23.110.68 - Running systemd command 2021-12-14 19:54:37,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl stop couchbase-server.service 2021-12-14 19:54:39,696 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:474] Couchbase stopped on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-14 19:54:39,696 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-14 19:54:39,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-14 19:54:39,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:39,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-12-14 19:54:39,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-12-14 19:54:39,796 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-14 19:54:39,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:54:40,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-14 19:54:40,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:54:40,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-14 19:54:40,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-14 19:54:40,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-14 19:54:40,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-14 19:54:40,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-14 19:54:40,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-14 19:54:40,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-14 19:54:40,431 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:716] 172.23.110.68 - Checking if couchbase-server is installed 2021-12-14 19:54:40,459 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:749] 172.23.110.68 - The version file /opt/couchbase/ VERSION.txt exists 2021-12-14 19:54:40,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /opt/couchbase/VERSION.txt 2021-12-14 19:54:40,530 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3906] 172.23.110.68 - Running systemd command 2021-12-14 19:54:40,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: systemctl start couchbase-server.service 2021-12-14 19:54:40,565 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:488] Couchbase started on ip:172.23.110.68 port:8091 ssh_username:root 2021-12-14 19:54:40,565 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x49d5def9, 0.0.0.0/0.0.0.0:59988] timeout=300.0>) 2021-12-14 19:54:40,571 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:40,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:40,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xd6b570bc, 0.0.0.0/0.0.0.0:38532] timeout=300.0>) 2021-12-14 19:54:42,585 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:42,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:42,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xe81db03b, 0.0.0.0/0.0.0.0:48082] timeout=300.0>) 2021-12-14 19:54:44,599 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:44,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:44,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x4c8fbd5c, 0.0.0.0/0.0.0.0:37012] timeout=300.0>) 2021-12-14 19:54:46,618 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:46,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:46,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.110.68:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x02075228, 0.0.0.0/0.0.0.0:57338] timeout=300.0>) 2021-12-14 19:54:48,641 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.68:8091/nodes/self error [Errno 111] Connection refused 2021-12-14 19:54:48,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L842 2021-12-14 19:54:48,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:54:50,676 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 360secs 2021-12-14 19:54:50,691 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-14 19:54:50,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L562 2021-12-14 19:54:50,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-14 19:55:00,720 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:55:00,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:01,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:55:01,177 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:01,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:55:01,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:55:01,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:55:01,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:55:01,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:55:01,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:55:01,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:55:01,342 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 19:55:01,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:01,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 19:55:01,884 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:01,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 19:55:01,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 19:55:01,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 19:55:01,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 19:55:01,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 19:55:01,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 19:55:01,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 19:55:02,013 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 19:55:02,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:02,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 19:55:02,546 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:02,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 19:55:02,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 19:55:02,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 19:55:02,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 19:55:02,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 19:55:02,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 19:55:02,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 19:55:02,674 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-14 19:55:02,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:03,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-14 19:55:03,171 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:03,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-14 19:55:03,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-14 19:55:03,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-14 19:55:03,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-14 19:55:03,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-14 19:55:03,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-14 19:55:03,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-14 19:55:03,289 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-14 19:55:03,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:03,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-14 19:55:03,796 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:03,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-14 19:55:03,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-14 19:55:03,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-14 19:55:03,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-14 19:55:03,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-14 19:55:03,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-14 19:55:03,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-14 19:55:03,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: rm -rf /data/* 2021-12-14 19:55:03,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: chown -R couchbase:couchbase /data 2021-12-14 19:55:04,019 | test | DEBUG | MainThread | [rest_client:set_data_path:2247] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-14 19:55:06,798 | test | DEBUG | MainThread | [rest_client:set_data_path:2250] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-14 19:55:06,799 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.64_8091 2021-12-14 19:55:06,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: rm -rf /data/* 2021-12-14 19:55:06,801 | infra | INFO | pool-3-thread-1 | [task:start_task:90] Thread 'node_init_task_172.23.110.64_8091' started 2021-12-14 19:55:06,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: chown -R couchbase:couchbase /data 2021-12-14 19:55:06,826 | infra | DEBUG | pool-3-thread-1 | [task:wait_until:134] Wait for expected condition to get satisfied 2021-12-14 19:55:06,835 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:06,835 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-14 19:55:06,864 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:06,865 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-14 19:55:06,865 | test | DEBUG | MainThread | [rest_client:set_data_path:2247] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-14 19:55:06,904 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:06,904 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-14 19:55:06,960 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:06,960 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-14 19:55:07,058 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:07,059 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-14 19:55:07,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:07,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-14 19:55:07,571 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:07,572 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-14 19:55:08,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:08,233 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-14 19:55:09,335 | test | DEBUG | MainThread | [rest_client:set_data_path:2250] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-14 19:55:09,335 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.65_8091 2021-12-14 19:55:09,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: rm -rf /data/* 2021-12-14 19:55:09,336 | infra | INFO | pool-3-thread-2 | [task:start_task:90] Thread 'node_init_task_172.23.110.65_8091' started 2021-12-14 19:55:09,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: chown -R couchbase:couchbase /data 2021-12-14 19:55:09,367 | infra | DEBUG | pool-3-thread-2 | [task:wait_until:134] Wait for expected condition to get satisfied 2021-12-14 19:55:09,371 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,371 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-14 19:55:09,405 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,407 | test | DEBUG | MainThread | [rest_client:set_data_path:2247] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-14 19:55:09,407 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-14 19:55:09,447 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,447 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-14 19:55:09,506 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,507 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-14 19:55:09,533 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,535 | test | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-14 19:55:09,609 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,611 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-14 19:55:09,793 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:09,795 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-14 19:55:10,137 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:10,138 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-14 19:55:10,799 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:10,799 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-14 19:55:12,102 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,104 | test | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-14 19:55:12,114 | test | DEBUG | pool-3-thread-1 | [task:call:6102] server: ip:172.23.110.64 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266614169600L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.64:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.64', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1913-enterprise', 'indexMemoryQuota': 512, 'uptime': u'86', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-14 19:55:12,117 | test | DEBUG | pool-3-thread-1 | [rest_client:set_service_mem_quota:939] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-14 19:55:12,203 | test | DEBUG | pool-3-thread-1 | [rest_client:set_indexer_storage_mode:963] settings/indexes params: storageMode=plasma 2021-12-14 19:55:12,223 | test | DEBUG | pool-3-thread-1 | [rest_client:init_cluster:850] settings/web params on 172.23.110.64:8091:password=password&port=8091&username=Administrator 2021-12-14 19:55:12,457 | test | DEBUG | MainThread | [rest_client:set_data_path:2250] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-14 19:55:12,459 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.66_8091 2021-12-14 19:55:12,460 | infra | INFO | pool-3-thread-3 | [task:start_task:90] Thread 'node_init_task_172.23.110.66_8091' started 2021-12-14 19:55:12,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: rm -rf /data/* 2021-12-14 19:55:12,483 | infra | DEBUG | pool-3-thread-3 | [task:wait_until:134] Wait for expected condition to get satisfied 2021-12-14 19:55:12,484 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,486 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-14 19:55:12,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: chown -R couchbase:couchbase /data 2021-12-14 19:55:12,507 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,507 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-14 19:55:12,519 | test | DEBUG | MainThread | [rest_client:set_data_path:2247] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-14 19:55:12,542 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,542 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-14 19:55:12,595 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,595 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-14 19:55:12,691 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,693 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-14 19:55:12,868 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:12,868 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-14 19:55:13,203 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:13,203 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-14 19:55:13,857 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:13,857 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-14 19:55:14,677 | test | DEBUG | pool-3-thread-2 | [task:call:6102] server: ip:172.23.110.65 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266813075456L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.65:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.65', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1913-enterprise', 'indexMemoryQuota': 512, 'uptime': u'73', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-14 19:55:14,678 | test | DEBUG | pool-3-thread-2 | [rest_client:set_service_mem_quota:939] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-14 19:55:14,690 | test | DEBUG | pool-3-thread-2 | [rest_client:set_indexer_storage_mode:963] settings/indexes params: storageMode=plasma 2021-12-14 19:55:14,704 | test | DEBUG | pool-3-thread-2 | [rest_client:init_cluster:850] settings/web params on 172.23.110.65:8091:password=password&port=8091&username=Administrator 2021-12-14 19:55:15,151 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,153 | test | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-14 19:55:15,414 | test | DEBUG | MainThread | [rest_client:set_data_path:2250] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-14 19:55:15,414 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.67_8091 2021-12-14 19:55:15,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: rm -rf /data/* 2021-12-14 19:55:15,417 | infra | INFO | pool-3-thread-4 | [task:start_task:90] Thread 'node_init_task_172.23.110.67_8091' started 2021-12-14 19:55:15,437 | infra | DEBUG | pool-3-thread-4 | [task:wait_until:134] Wait for expected condition to get satisfied 2021-12-14 19:55:15,440 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,441 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-14 19:55:15,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: chown -R couchbase:couchbase /data 2021-12-14 19:55:15,467 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,467 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-14 19:55:15,476 | test | DEBUG | MainThread | [rest_client:set_data_path:2247] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-12-14 19:55:15,506 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,506 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-14 19:55:15,561 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,561 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-14 19:55:15,655 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,655 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-14 19:55:15,828 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:15,829 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-14 19:55:16,164 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:16,164 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-14 19:55:16,819 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:16,819 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-14 19:55:17,724 | test | DEBUG | pool-3-thread-3 | [task:call:6102] server: ip:172.23.110.66 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266782978048L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.66:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.66', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1913-enterprise', 'indexMemoryQuota': 512, 'uptime': u'62', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-14 19:55:17,726 | test | DEBUG | pool-3-thread-3 | [rest_client:set_service_mem_quota:939] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-14 19:55:17,736 | test | DEBUG | pool-3-thread-3 | [rest_client:set_indexer_storage_mode:963] settings/indexes params: storageMode=plasma 2021-12-14 19:55:17,756 | test | DEBUG | pool-3-thread-3 | [rest_client:init_cluster:850] settings/web params on 172.23.110.66:8091:password=password&port=8091&username=Administrator 2021-12-14 19:55:18,002 | test | DEBUG | MainThread | [rest_client:set_data_path:2250] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-12-14 19:55:18,003 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: node_init_task_172.23.110.68_8091 2021-12-14 19:55:18,003 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.64_8091 2021-12-14 19:55:18,003 | infra | INFO | pool-3-thread-5 | [task:start_task:90] Thread 'node_init_task_172.23.110.68_8091' started 2021-12-14 19:55:18,005 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.65_8091 2021-12-14 19:55:18,005 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.66_8091 2021-12-14 19:55:18,006 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.67_8091 2021-12-14 19:55:18,026 | infra | DEBUG | pool-3-thread-5 | [task:wait_until:134] Wait for expected condition to get satisfied 2021-12-14 19:55:18,029 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,029 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.01 seconds. Reason: None 2021-12-14 19:55:18,052 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,053 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.02 seconds. Reason: None 2021-12-14 19:55:18,088 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,088 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.04 seconds. Reason: None 2021-12-14 19:55:18,115 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,115 | test | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-14 19:55:18,144 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,144 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.08 seconds. Reason: None 2021-12-14 19:55:18,239 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,240 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.16 seconds. Reason: None 2021-12-14 19:55:18,414 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,414 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.32 seconds. Reason: None 2021-12-14 19:55:18,747 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:18,749 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 0.64 seconds. Reason: None 2021-12-14 19:55:19,404 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:19,404 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 1.28 seconds. Reason: None 2021-12-14 19:55:20,688 | test | DEBUG | pool-3-thread-4 | [task:call:6102] server: ip:172.23.110.67 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266794680320L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.67:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.67', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1913-enterprise', 'indexMemoryQuota': 512, 'uptime': u'50', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-14 19:55:20,690 | test | DEBUG | pool-3-thread-4 | [rest_client:set_service_mem_quota:939] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-14 19:55:20,700 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> wait_until():L139 2021-12-14 19:55:20,700 | test | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 2.56 seconds. Reason: None 2021-12-14 19:55:20,703 | test | DEBUG | pool-3-thread-4 | [rest_client:set_indexer_storage_mode:963] settings/indexes params: storageMode=plasma 2021-12-14 19:55:20,721 | test | DEBUG | pool-3-thread-4 | [rest_client:init_cluster:850] settings/web params on 172.23.110.67:8091:password=password&port=8091&username=Administrator 2021-12-14 19:55:20,859 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for node_init_task_172.23.110.68_8091 2021-12-14 19:55:23,273 | test | DEBUG | pool-3-thread-5 | [task:call:6102] server: ip:172.23.110.68 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 266848759808L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 21490, 'hostname': u'172.23.110.68:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458753, 'id': u'ns_1@cb.local', 'mcdMemoryReserved': 205895, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-pc-linux-gnu', 'ip': u'172.23.110.68', 'curr_items': 0, 'mcdMemoryAllocated': 205895, 'services': [u'kv'], 'version': u'7.1.0-1913-enterprise', 'indexMemoryQuota': 512, 'uptime': u'40', 'clusterMembership': u'active', 'storageTotalRam': 257368L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 269870723072L, 'memoryQuota': 153755, 'status': u'healthy'} 2021-12-14 19:55:23,275 | test | DEBUG | pool-3-thread-5 | [rest_client:set_service_mem_quota:939] Setting RAM quotas: {'memoryQuota': 185215} 2021-12-14 19:55:23,286 | test | DEBUG | pool-3-thread-5 | [rest_client:set_indexer_storage_mode:963] settings/indexes params: storageMode=plasma 2021-12-14 19:55:23,302 | test | DEBUG | pool-3-thread-5 | [rest_client:init_cluster:850] settings/web params on 172.23.110.68:8091:password=password&port=8091&username=Administrator 2021-12-14 19:55:23,433 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:742] RAM quota was defined less than 100 MB: 2021-12-14 19:55:23,433 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:745] Node: 172.23.110.64: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 255993136 kB\n', 'MemAvailable: 259571964 kB\n', 'Buffers: 40 kB\n', 'Cached: 4364820 kB\n', 'SwapCached: 372 kB\n', 'Active: 1496260 kB\n', 'Inactive: 3310352 kB\n', 'Active(anon): 424844 kB\n', 'Inactive(anon): 365936 kB\n', 'Active(file): 1071416 kB\n', 'Inactive(file): 2944416 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4151664 kB\n', 'Dirty: 80 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 441796 kB\n', 'Mapped: 115280 kB\n', 'Shmem: 349008 kB\n', 'Slab: 921856 kB\n', 'SReclaimable: 447052 kB\n', 'SUnreclaim: 474804 kB\n', 'KernelStack: 27584 kB\n', 'PageTables: 11704 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3113156 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 273408 kB\n', 'DirectMap2M: 10870784 kB\n', 'DirectMap1G: 258998272 kB\n'] 2021-12-14 19:55:23,434 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:745] Node: 172.23.110.65: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256170372 kB\n', 'MemAvailable: 259593376 kB\n', 'Buffers: 40 kB\n', 'Cached: 4387544 kB\n', 'SwapCached: 384 kB\n', 'Active: 1492040 kB\n', 'Inactive: 3357244 kB\n', 'Active(anon): 431524 kB\n', 'Inactive(anon): 415724 kB\n', 'Active(file): 1060516 kB\n', 'Inactive(file): 2941520 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4173732 kB\n', 'Dirty: 24 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 461260 kB\n', 'Mapped: 101564 kB\n', 'Shmem: 385532 kB\n', 'Slab: 705528 kB\n', 'SReclaimable: 305024 kB\n', 'SUnreclaim: 400504 kB\n', 'KernelStack: 27456 kB\n', 'PageTables: 12344 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3160452 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 17176576 kB\n', 'DirectMap1G: 252706816 kB\n'] 2021-12-14 19:55:23,434 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:745] Node: 172.23.110.66: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256082908 kB\n', 'MemAvailable: 259517048 kB\n', 'Buffers: 28 kB\n', 'Cached: 4446436 kB\n', 'SwapCached: 684 kB\n', 'Active: 1493968 kB\n', 'Inactive: 3399704 kB\n', 'Active(anon): 440160 kB\n', 'Inactive(anon): 454292 kB\n', 'Active(file): 1053808 kB\n', 'Inactive(file): 2945412 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4092780 kB\n', 'Dirty: 40 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 447492 kB\n', 'Mapped: 106656 kB\n', 'Shmem: 447236 kB\n', 'Slab: 740376 kB\n', 'SReclaimable: 319052 kB\n', 'SUnreclaim: 421324 kB\n', 'KernelStack: 27536 kB\n', 'PageTables: 11732 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3298904 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 779660 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 0 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 16115712 kB\n', 'DirectMap1G: 253755392 kB\n'] 2021-12-14 19:55:23,434 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:745] Node: 172.23.110.67: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256256120 kB\n', 'MemAvailable: 259804160 kB\n', 'Buffers: 40 kB\n', 'Cached: 4284588 kB\n', 'SwapCached: 124 kB\n', 'Active: 1566724 kB\n', 'Inactive: 3167708 kB\n', 'Active(anon): 479952 kB\n', 'Inactive(anon): 168284 kB\n', 'Active(file): 1086772 kB\n', 'Inactive(file): 2999424 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4180008 kB\n', 'Dirty: 484 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 450988 kB\n', 'Mapped: 108508 kB\n', 'Shmem: 198416 kB\n', 'Slab: 730632 kB\n', 'SReclaimable: 345900 kB\n', 'SUnreclaim: 384732 kB\n', 'KernelStack: 27472 kB\n', 'PageTables: 11024 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 2989172 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 8192 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 259072 kB\n', 'DirectMap2M: 21370880 kB\n', 'DirectMap1G: 248512512 kB\n'] 2021-12-14 19:55:23,434 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:745] Node: 172.23.110.68: RAM: ['MemTotal: 263545628 kB\n', 'MemFree: 256345980 kB\n', 'MemAvailable: 259769908 kB\n', 'Buffers: 40 kB\n', 'Cached: 4247504 kB\n', 'SwapCached: 1260 kB\n', 'Active: 1658408 kB\n', 'Inactive: 3047972 kB\n', 'Active(anon): 582180 kB\n', 'Inactive(anon): 96764 kB\n', 'Active(file): 1076228 kB\n', 'Inactive(file): 2951208 kB\n', 'Unevictable: 0 kB\n', 'Mlocked: 0 kB\n', 'SwapTotal: 4194300 kB\n', 'SwapFree: 4095276 kB\n', 'Dirty: 804 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 457004 kB\n', 'Mapped: 102392 kB\n', 'Shmem: 220076 kB\n', 'Slab: 680000 kB\n', 'SReclaimable: 280624 kB\n', 'SUnreclaim: 399376 kB\n', 'KernelStack: 27312 kB\n', 'PageTables: 12844 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 135967112 kB\n', 'Committed_AS: 3087368 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 778636 kB\n', 'VmallocChunk: 34224705532 kB\n', 'Percpu: 64448 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 271360 kB\n', 'DirectMap2M: 18212864 kB\n', 'DirectMap1G: 251658240 kB\n'] 2021-12-14 19:55:23,436 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 19:55:23,436 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 19:55:23,436 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 19:55:23,437 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-14 19:55:23,437 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-14 19:55:23,441 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:55:23,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:23,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:55:23,913 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:23,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:55:23,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:55:23,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:55:23,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:55:24,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:55:24,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:55:24,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:55:24,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-14 19:55:24,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.64 port:8091 ssh_username:root: [] 2021-12-14 19:55:24,084 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 19:55:24,084 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-14 19:55:24,088 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 19:55:24,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:24,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 19:55:24,605 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:24,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 19:55:24,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 19:55:24,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 19:55:24,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 19:55:24,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 19:55:24,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 19:55:24,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 19:55:24,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-14 19:55:24,757 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.65 port:8091 ssh_username:root: [] 2021-12-14 19:55:24,759 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 19:55:24,759 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-14 19:55:24,763 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 19:55:24,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:25,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 19:55:25,302 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:25,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 19:55:25,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 19:55:25,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 19:55:25,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 19:55:25,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 19:55:25,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 19:55:25,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 19:55:25,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-14 19:55:25,460 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.66 port:8091 ssh_username:root: [] 2021-12-14 19:55:25,460 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 19:55:25,461 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-14 19:55:25,466 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-14 19:55:25,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:25,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-14 19:55:25,930 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:25,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-14 19:55:25,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-14 19:55:25,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-14 19:55:25,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-14 19:55:26,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-14 19:55:26,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-14 19:55:26,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-14 19:55:26,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-14 19:55:26,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.67 port:8091 ssh_username:root: [] 2021-12-14 19:55:26,082 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-14 19:55:26,082 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-14 19:55:26,085 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-14 19:55:26,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:55:26,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-14 19:55:26,575 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:55:26,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-14 19:55:26,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-14 19:55:26,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-14 19:55:26,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-14 19:55:26,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-14 19:55:26,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-14 19:55:26,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-14 19:55:26,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-12-14 19:55:26,726 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:340] Current vBuckets on node ip:172.23.110.68 port:8091 ssh_username:root: [] 2021-12-14 19:55:26,726 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-14 19:55:26,726 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:359] ========= CHANGED ENVIRONMENT SETTING =========== 2021-12-14 19:55:26,726 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:361] Wait for all the services to come up after change_env_vars update 2021-12-14 19:55:26,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L363 2021-12-14 19:55:26,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-14 19:55:36,736 | test | INFO | MainThread | [basetestcase:initialize_cluster:465] Cluster C1 initialized 2021-12-14 19:55:36,736 | test | INFO | MainThread | [basetestcase:log_setup_status:690] ========= BaseTestCase setup finished for test #1 SystemTestMagma ========= ############################################################ # # Step 1: Create a 3 node cluster # ############################################################ 2021-12-14 19:55:36,789 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1639540536.77 2021-12-14 19:55:36,790 | infra | INFO | pool-3-thread-6 | [task:start_task:90] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1639540536.77' started 2021-12-14 19:55:36,790 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1639540536.77 2021-12-14 19:55:36,792 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1184] Adding remote node 172.23.110.65:8091 to cluster 172.23.110.64:8091 2021-12-14 19:55:41,171 | test | DEBUG | pool-3-thread-6 | [rest_client:add_node:1184] Adding remote node 172.23.110.66:8091 to cluster 172.23.110.64:8091 2021-12-14 19:55:44,954 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-14 19:55:45,012 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-14 19:55:45,013 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 0.140318704066 | Cluster node | | 172.23.110.65 | None | | | <--- IN --- | | 172.23.110.66 | None | | | <--- IN --- | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-14 19:55:45,023 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-14 19:55:45,023 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-14 19:55:45,038 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-14 19:55:45,039 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-14 19:55:45,039 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-14 19:55:55,052 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-14 19:55:55,052 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-14 19:55:55,069 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-14 19:55:55,071 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-14 19:55:55,072 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-14 19:56:05,086 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-14 19:56:05,086 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-14 19:56:10,098 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: none, progress: 100 2021-12-14 19:56:10,112 | test | INFO | pool-3-thread-6 | [task:check:444] Rebalance completed with progress: 100% in 25.0999999046 sec 2021-12-14 19:56:10,114 | infra | INFO | pool-3-thread-6 | [task:complete_task:105] Thread 'Rebalance_task_IN=[172.23.110.65,172.23.110.66]_OUT=[]_1639540536.77' completed 2021-12-14 19:56:10,174 | test | INFO | MainThread | [rest_client:set_metadata_purge_interval:2155] Successfully set Metadata Purge Interval to 0.04 ############################################################ # # Step 2: Create required buckets and collections. # ############################################################ 2021-12-14 19:56:10,176 | test | INFO | MainThread | [opd:create_required_buckets:49] Get the available memory quota 2021-12-14 19:56:10,201 | test | INFO | MainThread | [opd:create_required_buckets:57] Create CB buckets 2021-12-14 19:56:10,203 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1569] Creating bucket: GleamBookUsers0 2021-12-14 19:56:10,203 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: bucket_GleamBookUsers0_create_task 2021-12-14 19:56:10,203 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for bucket_GleamBookUsers0_create_task 2021-12-14 19:56:10,259 | infra | INFO | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:344] Creating 'membase' bucket GleamBookUsers0 2021-12-14 19:56:10,259 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:346] http://172.23.110.64:8091/pools/default/buckets with param: storageBackend=magma&fragmentationPercentage=50&replicaIndex=1&maxTTL=0&flushEnabled=1&durabilityMinLevel=none&compressionMode=passive&bucketType=membase&conflictResolutionType=seqno&name=GleamBookUsers0&replicaNumber=1&ramQuotaMB=102400&evictionPolicy=fullEviction 2021-12-14 19:56:10,296 | infra | DEBUG | pool-3-thread-7 | [BucketOperations_Rest:create_bucket:354] 0.04 seconds to create bucket GleamBookUsers0 2021-12-14 19:56:10,298 | infra | INFO | pool-3-thread-7 | [task:complete_task:105] Thread 'bucket_GleamBookUsers0_create_task' completed 2021-12-14 19:56:10,298 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1581] Wait for memcached to accept cbstats or any other bucket request connections 2021-12-14 19:56:10,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1583 2021-12-14 19:56:10,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-14 19:56:12,374 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-14 19:56:12,375 | test | INFO | MainThread | [opd:create_required_buckets:79] Rebalance Starts 2021-12-14 19:56:12,398 | test | DEBUG | MainThread | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-14 19:56:12,451 | test | DEBUG | MainThread | [rest_client:rebalance:1444] Rebalance operation started 2021-12-14 19:56:12,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-14 19:56:12,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-14 19:56:12,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-14 19:56:15,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-14 19:56:15,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-14 19:56:20,561 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-14 19:56:20,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-14 19:56:23,572 | test | INFO | MainThread | [rest_client:monitorRebalance:1584] Rebalance done. Taken 11.1199998856 seconds to complete 2021-12-14 19:56:23,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1587 2021-12-14 19:56:23,579 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-12-14 19:56:28,581 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection0 2021-12-14 19:56:28,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:28,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:29,144 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection1 2021-12-14 19:56:29,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:29,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:29,707 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection2 2021-12-14 19:56:29,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:29,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:30,275 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection3 2021-12-14 19:56:30,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:30,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:30,832 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection4 2021-12-14 19:56:30,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:30,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:31,391 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection5 2021-12-14 19:56:31,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:31,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:31,947 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection6 2021-12-14 19:56:32,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:32,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:32,509 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection7 2021-12-14 19:56:32,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:32,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:33,065 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection8 2021-12-14 19:56:33,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:33,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:33,631 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection9 2021-12-14 19:56:33,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:33,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:34,200 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection10 2021-12-14 19:56:34,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:34,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:34,760 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection11 2021-12-14 19:56:34,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:34,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:35,322 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection12 2021-12-14 19:56:35,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:35,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:35,878 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection13 2021-12-14 19:56:35,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:35,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:36,440 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection14 2021-12-14 19:56:36,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:36,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:37,003 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection15 2021-12-14 19:56:37,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:37,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:37,562 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection16 2021-12-14 19:56:37,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:37,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:38,121 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection17 2021-12-14 19:56:38,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:38,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:38,680 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection18 2021-12-14 19:56:38,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:38,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:39,243 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection19 2021-12-14 19:56:39,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:39,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:39,802 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection20 2021-12-14 19:56:39,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:39,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:40,365 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection21 2021-12-14 19:56:40,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:40,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:40,926 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection22 2021-12-14 19:56:40,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:40,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:41,486 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection23 2021-12-14 19:56:41,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:41,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:42,048 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection24 2021-12-14 19:56:42,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:42,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:42,611 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection25 2021-12-14 19:56:42,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:42,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:43,176 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection26 2021-12-14 19:56:43,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:43,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:43,739 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection27 2021-12-14 19:56:43,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:43,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:44,301 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection28 2021-12-14 19:56:44,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:44,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:44,864 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection29 2021-12-14 19:56:44,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:44,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:45,424 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection30 2021-12-14 19:56:45,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:45,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:45,987 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection31 2021-12-14 19:56:46,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:46,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:46,552 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection32 2021-12-14 19:56:46,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:46,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:47,125 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection33 2021-12-14 19:56:47,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:47,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:47,691 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection34 2021-12-14 19:56:47,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:47,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:48,262 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection35 2021-12-14 19:56:48,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:48,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:48,822 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection36 2021-12-14 19:56:48,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:48,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:49,384 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection37 2021-12-14 19:56:49,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:49,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:49,947 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection38 2021-12-14 19:56:50,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:50,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:50,506 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection39 2021-12-14 19:56:50,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:50,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:51,071 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection40 2021-12-14 19:56:51,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:51,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:51,634 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection41 2021-12-14 19:56:51,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:51,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:52,197 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection42 2021-12-14 19:56:52,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:52,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:52,759 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection43 2021-12-14 19:56:52,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:52,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:53,323 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection44 2021-12-14 19:56:53,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:53,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:53,888 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection45 2021-12-14 19:56:53,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:53,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:54,446 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection46 2021-12-14 19:56:54,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:54,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:55,016 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection47 2021-12-14 19:56:55,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:55,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:55,584 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection48 2021-12-14 19:56:55,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:55,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:56,154 | test | DEBUG | MainThread | [bucket_ready_functions:create_collection:948] Creating Collection GleamBookUsers0:_default:VolumeCollection49 2021-12-14 19:56:56,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> create_required_collections():L116 2021-12-14 19:56:56,219 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 0.5 seconds. Reason: None 2021-12-14 19:56:56,720 | test | DEBUG | MainThread | [opd:create_required_collections:119] Collections list == ['VolumeCollection46', 'VolumeCollection47', 'VolumeCollection44', 'VolumeCollection45', 'VolumeCollection48', 'VolumeCollection49', 'VolumeCollection42', 'VolumeCollection43', 'VolumeCollection40', 'VolumeCollection41', '_default', 'VolumeCollection35', 'VolumeCollection36', 'VolumeCollection33', 'VolumeCollection34', 'VolumeCollection39', 'VolumeCollection37', 'VolumeCollection38', 'VolumeCollection7', 'VolumeCollection6', 'VolumeCollection5', 'VolumeCollection4', 'VolumeCollection3', 'VolumeCollection31', 'VolumeCollection2', 'VolumeCollection32', 'VolumeCollection1', 'VolumeCollection0', 'VolumeCollection30', 'VolumeCollection9', 'VolumeCollection8', 'VolumeCollection19', 'VolumeCollection24', 'VolumeCollection25', 'VolumeCollection22', 'VolumeCollection23', 'VolumeCollection28', 'VolumeCollection29', 'VolumeCollection26', 'VolumeCollection27', 'VolumeCollection20', 'VolumeCollection21', 'VolumeCollection13', 'VolumeCollection14', 'VolumeCollection11', 'VolumeCollection12', 'VolumeCollection17', 'VolumeCollection18', 'VolumeCollection15', 'VolumeCollection16', 'VolumeCollection10'] 2021-12-14 19:56:56,733 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:2049] settings/autoFailover params: failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false [ip:172.23.110.67 port:8091 ssh_username:root, ip:172.23.110.68 port:8091 ssh_username:root] 2021-12-14 19:56:56,769 | infra | INFO | MainThread | [BucketOperations_Rest:update_memcached_settings:373] Updating memcached properties 2021-12-14 19:56:56,769 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:374] http://172.23.110.64:8091/pools/default/settings/memcached/global with param: num_reader_threads=disk_io_optimized&num_writer_threads=disk_io_optimized&num_storage_threads=40 2021-12-14 19:56:56,792 | infra | DEBUG | MainThread | [BucketOperations_Rest:update_memcached_settings:380] Memcached settings updated ############################################################ # # Step 1: Create 40000000 items sequentially # ############################################################ 2021-12-14 19:56:56,796 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 19:56:56,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 19:56:57,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 19:56:57,273 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 19:56:57,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 19:56:57,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 19:56:57,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 19:56:57,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 19:56:57,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 19:56:57,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 19:56:57,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 19:56:57,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2021-12-14 19:56:57,515 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 ############################################################ # # Step 2: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 959.51171875 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 40000000 Final Start: 0 Final End: 40000000 2021-12-14 19:56:58,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:56:58,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1639540619.08 2021-12-14 19:56:59,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:56:59,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1639540620.11 2021-12-14 19:57:00,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:00,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1639540621.13 2021-12-14 19:57:01,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:01,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1639540622.15 2021-12-14 19:57:02,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:02,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1639540623.17 2021-12-14 19:57:03,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:03,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1639540624.2 2021-12-14 19:57:04,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:04,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1639540625.22 2021-12-14 19:57:05,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:05,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1639540626.23 2021-12-14 19:57:06,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:06,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1639540627.25 2021-12-14 19:57:07,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:07,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1639540628.26 2021-12-14 19:57:08,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:08,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:09,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:09,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:10,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:10,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:11,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:11,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:12,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:12,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:13,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:13,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:14,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:14,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:15,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:15,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:16,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:16,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:17,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:17,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:18,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:18,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:19,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:19,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:20,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:20,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:21,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:21,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:22,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:22,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:23,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:23,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:24,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:24,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:25,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:25,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:27,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:27,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:28,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:28,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:29,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:29,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:30,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:30,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:31,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:31,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:32,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:32,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:33,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:33,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:34,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:34,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:35,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:35,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:36,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:36,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:37,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:37,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:38,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:38,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:39,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:39,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:40,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:40,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:41,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:41,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:42,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:42,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:43,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:43,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:44,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:44,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:45,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:45,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:46,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:46,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:47,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:47,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 19:57:48,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 19:57:48,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection42_4_1639540625.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1639540629.61 Loader_GleamBookUsers0__default_VolumeCollection47_9_1639540620.11 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639540630.62 Loader_GleamBookUsers0__default_VolumeCollection43_3_1639540626.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639540631.63 Loader_GleamBookUsers0__default_VolumeCollection45_7_1639540622.15 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639540632.65 Loader_GleamBookUsers0__default_VolumeCollection41_1_1639540628.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639540633.67 Loader_GleamBookUsers0__default_VolumeCollection49_5_1639540624.2 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639540634.69 Loader_GleamBookUsers0__default_VolumeCollection44_8_1639540621.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639540635.7 Loader_GleamBookUsers0__default_VolumeCollection40_2_1639540627.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639540636.72 Loader_GleamBookUsers0__default_VolumeCollection46_10_1639540619.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639540637.73 Loader_GleamBookUsers0__default_VolumeCollection48_6_1639540623.17 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639540638.75 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639540630.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639540639.76 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639540632.65 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639540640.83 Loader_GleamBookUsers0__default_VolumeCollection35_0_1639540629.61 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639540641.93 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639540631.63 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639540642.94 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639540634.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639540643.96 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639540636.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639540644.97 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639540633.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639540645.98 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639540638.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639540647.0 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639540635.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639540648.01 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639540637.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639540649.03 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639540639.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639540650.04 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639540640.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639540651.06 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639540643.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639540652.07 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639540645.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639540653.08 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639540642.94 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639540654.42 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639540648.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639540655.44 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639540641.93 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639540656.45 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639540644.97 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639540657.47 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639540647.0 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639540658.48 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639540649.03 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639540659.5 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639540650.04 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639540660.51 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639540652.07 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639540661.52 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639540651.06 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639540662.54 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639540654.42 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639540663.55 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639540656.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639540664.57 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639540653.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639540665.58 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639540657.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639540666.6 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639540655.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639540667.62 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639540658.48 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639540668.63 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639540659.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639540669.64 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639540660.51 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639540661.52 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639540662.54 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639540663.55 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639540664.57 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639540665.58 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639540667.62 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639540666.6 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639540668.63 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639540669.64 is completed! 2021-12-14 23:49:23,828 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-14 23:49:23,832 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 23:49:23,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:24,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 23:49:24,282 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:24,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 23:49:24,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 23:49:24,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 23:49:24,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 23:49:24,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 23:49:24,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 23:49:24,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 23:49:24,410 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-14 23:49:24,411 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554564.41 2021-12-14 23:49:24,413 | infra | INFO | pool-3-thread-8 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554564.41' started 2021-12-14 23:49:24,414 | infra | INFO | pool-3-thread-8 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554564.41' completed 2021-12-14 23:49:24,417 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 23:49:24,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:24,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 23:49:24,960 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:24,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 23:49:24,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 23:49:25,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 23:49:25,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 23:49:25,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 23:49:25,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 23:49:25,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 23:49:25,127 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-14 23:49:25,128 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.13 2021-12-14 23:49:25,130 | infra | INFO | pool-3-thread-9 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.13' started 2021-12-14 23:49:25,131 | infra | INFO | pool-3-thread-9 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.13' completed 2021-12-14 23:49:25,132 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 23:49:25,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:25,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 23:49:25,663 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:25,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 23:49:25,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 23:49:25,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 23:49:25,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 23:49:25,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 23:49:25,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 23:49:25,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 23:49:25,790 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-14 23:49:25,790 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.79 2021-12-14 23:49:25,792 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554564.41 2021-12-14 23:49:25,792 | infra | INFO | pool-3-thread-10 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.79' started 2021-12-14 23:49:25,792 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 23:49:25,792 | infra | INFO | pool-3-thread-10 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.79' completed 2021-12-14 23:49:25,792 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.13 2021-12-14 23:49:25,793 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 23:49:25,795 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639554565.79 2021-12-14 23:49:25,795 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 23:49:25,838 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 19992.5, u'diskFetches': 0, u'dataUsed': 4508126063170L, u'itemCount': 1999960126, u'quotaPercentUsed': 78.39562822133303, u'storageTotals': {u'hdd': {u'usedByData': 4553875343615L, u'used': 4866615740985L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 31109194249671L}, u'ram': {u'usedByData': 252529994520L, u'used': 330471776256L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 4553875343615L, u'vbActiveNumNonResident': 1860438313, u'memUsed': 252529994520L} 2021-12-14 23:49:26,220 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 1999960126 | 300.00 GiB | 235.19 GiB | 4.14 TiB | 6.97622973509 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-14 23:49:26,220 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 0 | 2000000000 | 0-0 | 0-40000000 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-14 23:49:26,589 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.97622973509 2021-12-14 23:49:26,592 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 23:49:26,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:26,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 23:49:27,075 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:27,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 23:49:27,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 23:49:27,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 23:49:27,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 23:49:27,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 23:49:27,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 23:49:27,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 23:49:27,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-14 23:49:27,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-14 23:49:27,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-14 23:49:27,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-14 23:49:28,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-14 23:49:29,232 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 23:49:29,236 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 23:49:29,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:29,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 23:49:29,733 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:29,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 23:49:29,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 23:49:29,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 23:49:29,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 23:49:29,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 23:49:29,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 23:49:29,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 23:49:29,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-14 23:49:30,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-14 23:49:30,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-14 23:49:30,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-14 23:49:30,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-14 23:49:31,801 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 23:49:31,805 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 23:49:31,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:32,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 23:49:32,315 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:32,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 23:49:32,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 23:49:32,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 23:49:32,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 23:49:32,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 23:49:32,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 23:49:32,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 23:49:32,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-14 23:49:32,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-14 23:49:32,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-14 23:49:32,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-14 23:49:33,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-14 23:49:34,354 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 23:49:34,354 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 4430322MB 2021-12-14 23:49:34,355 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6941MB 2021-12-14 23:49:34,355 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 207407MB 2021-12-14 23:49:34,355 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 4222525MB 2021-12-14 23:49:34,358 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 23:49:34,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:49:34,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 23:49:34,861 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:49:34,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 23:49:34,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 23:49:34,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 23:49:34,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 23:49:34,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 23:49:34,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 23:49:34,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 23:49:35,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-14 23:49:35,082 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-14 23:49:35,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-14 23:49:35,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-14 23:49:36,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-14 23:49:36,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-14 23:49:36,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-14 23:49:37,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-14 23:49:37,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-14 23:49:38,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-14 23:49:38,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-14 23:49:39,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-14 23:49:39,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-14 23:49:39,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-14 23:49:40,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-14 23:49:40,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-14 23:49:41,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-14 23:49:41,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-14 23:49:42,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-14 23:49:42,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-14 23:49:42,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-14 23:49:43,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-14 23:49:43,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-14 23:49:44,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-14 23:49:44,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-14 23:49:45,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-14 23:49:45,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-14 23:49:46,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-14 23:49:46,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-14 23:49:46,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-14 23:49:47,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-14 23:49:47,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-14 23:49:47,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-14 23:49:48,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-14 23:49:48,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-14 23:49:49,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-14 23:49:49,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-14 23:49:50,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-14 23:49:50,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-14 23:49:51,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-14 23:49:51,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-14 23:49:52,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-14 23:49:52,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-14 23:49:52,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-14 23:49:53,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-14 23:49:53,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-14 23:49:54,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-14 23:49:54,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-14 23:49:55,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-14 23:49:55,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-14 23:49:55,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-14 23:49:56,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-14 23:49:56,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-14 23:49:57,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-14 23:49:57,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-14 23:49:57,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-14 23:49:58,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-14 23:49:58,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-14 23:49:59,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-14 23:49:59,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-14 23:49:59,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-14 23:50:00,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-14 23:50:00,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-14 23:50:01,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-14 23:50:01,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-14 23:50:01,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-14 23:50:02,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 23:50:02,375 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 23:50:02,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:50:02,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 23:50:02,917 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:50:02,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 23:50:02,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 23:50:02,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 23:50:02,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 23:50:03,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 23:50:03,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 23:50:03,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 23:50:03,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-14 23:50:03,141 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-14 23:50:03,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-14 23:50:03,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-14 23:50:04,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-14 23:50:04,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-14 23:50:05,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-14 23:50:05,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-14 23:50:05,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-14 23:50:06,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-14 23:50:06,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-14 23:50:07,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-14 23:50:07,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-14 23:50:08,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-14 23:50:08,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-14 23:50:09,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-14 23:50:09,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-14 23:50:09,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-14 23:50:10,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-14 23:50:10,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-14 23:50:11,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-14 23:50:11,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-14 23:50:12,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-14 23:50:12,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-14 23:50:13,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-14 23:50:13,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-14 23:50:14,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-14 23:50:14,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-14 23:50:15,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-14 23:50:15,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-14 23:50:15,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-14 23:50:16,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-14 23:50:16,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-14 23:50:17,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-14 23:50:17,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-14 23:50:18,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-14 23:50:18,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-14 23:50:19,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-14 23:50:19,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-14 23:50:19,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-14 23:50:20,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-14 23:50:20,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-14 23:50:21,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-14 23:50:21,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-14 23:50:21,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-14 23:50:22,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-14 23:50:22,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-14 23:50:23,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-14 23:50:23,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-14 23:50:24,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-14 23:50:24,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-14 23:50:25,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-14 23:50:25,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-14 23:50:25,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-14 23:50:26,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-14 23:50:26,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-14 23:50:27,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-14 23:50:27,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-14 23:50:28,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-14 23:50:28,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-14 23:50:29,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-14 23:50:29,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-14 23:50:30,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-14 23:50:30,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-14 23:50:30,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-14 23:50:31,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-14 23:50:31,582 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-14 23:50:31,586 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 23:50:31,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:50:31,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 23:50:32,065 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:50:32,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 23:50:32,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 23:50:32,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 23:50:32,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 23:50:32,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 23:50:32,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 23:50:32,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 23:50:32,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-14 23:50:32,278 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-14 23:50:32,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-14 23:50:32,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-14 23:50:33,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-14 23:50:33,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-14 23:50:34,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-14 23:50:34,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-14 23:50:35,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-14 23:50:35,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-14 23:50:36,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-14 23:50:36,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-14 23:50:37,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-14 23:50:37,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-14 23:50:37,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-14 23:50:38,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-14 23:50:38,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-14 23:50:39,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-14 23:50:39,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-14 23:50:40,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-14 23:50:40,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-14 23:50:41,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-14 23:50:41,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-14 23:50:42,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-14 23:50:42,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-14 23:50:43,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-14 23:50:43,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-14 23:50:44,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-14 23:50:44,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-14 23:50:45,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-14 23:50:45,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-14 23:50:45,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-14 23:50:46,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-14 23:50:46,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-14 23:50:47,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-14 23:50:47,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-14 23:50:48,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-14 23:50:48,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-14 23:50:49,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-14 23:50:49,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-14 23:50:50,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-14 23:50:50,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-14 23:50:51,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-14 23:50:51,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-14 23:50:52,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-14 23:50:52,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-14 23:50:53,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-14 23:50:53,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-14 23:50:54,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-14 23:50:54,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-14 23:50:54,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-14 23:50:55,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-14 23:50:55,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-14 23:50:56,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-14 23:50:56,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-14 23:50:56,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-14 23:50:57,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-14 23:50:57,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-14 23:50:58,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-14 23:50:58,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-14 23:50:59,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-14 23:50:59,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-14 23:51:00,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-14 23:51:00,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-14 23:51:01,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-14 23:51:01,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-14 23:51:01,944 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 23:51:01,944 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 23171.333984375, u'ReplayWALOffset': u'2852:5562368', u'NTablesCreated': 152107, u'RecentReadIOAmpSet': 0.0207111444324255, u'ReadIOAmp': 0.33040159940719604, u'BlockCacheMisses': 5375464, u'NFlushes': 50614, u'RecentWriteAmp': 4.421045303344727, u'NumSyncFlushes': 0, u'TotalMemUsed': 92489713, u'NBloomFilterHits': 290461937, u'Fragmentation': 0.0020469108130782843, u'NumNormalFlushes': 26026, u'RecentBlockCacheHitRatio': 35.4052848815918, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 47513, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 2515992, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 166303758, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 101219, u'NReadBytes': 0, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 229587376, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 0}, {u'TargetSize': 11728633856L, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 0}, {u'TargetSize': 45917476066L, u'LogicalSize': 23457267712L, u'SizeRatio': 0.5108570793020817, u'NWriteBytes': 23457267712L, u'BloomFilterMemUsed': 0, u'NumTables': 555, u'WastedSpace': 498529679, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 23457267712L}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 141055650, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58560, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 0, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 172020, u'CompactWriter': {u'NDataBlockWrites': 0, u'NWriteBytes': 0, u'NIndexBlockWrites': 0, u'NBytesIncoming': 0, u'NSyncs': 0, u'NInserts': 0, u'NWriteIOs': 0}, u'NWriteBytes': 23495603681L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 23457267712L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7585106748409558, u'BufferMemUsed': 482326, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 23457267712L, u'NTableFiles': 555, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 151280, u'TreeSnapshotMemoryUsed': 100800, u'TotalDiskUsage': 27683450880L, u'BytesIncoming': 30975969700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 0, u'NItemsPut': 21426350, u'NSyncs': 24588, u'CompactReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'ActiveDiskUsage': 23457267712L, u'NTablesDeleted': 0, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 7157256, u'NWriteBytes': 23495603681L, u'NIndexBlockWrites': 161877, u'NBytesIncoming': 31166427067L, u'NSyncs': 24588, u'NInserts': 21424714, u'NWriteIOs': 101219}}, u'NReadBytes': 22549191350L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 240.39999389648438, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 153072619, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5057, u'NTxns': 2712378, u'NWriteBytes': 38814138368L, u'NSegments': 2852, u'NSyncs': 2712378, u'NLoggedBytes': 38814138368L, u'NActiveSegments': 1, u'NWrites': 2712532, u'NInserts': 45254084, u'DiskUsed': 11481088}, u'NGets': 9742501, u'NReadBytesSet': 1519524761, u'NReadBytesCompact': 11735540951L, u'IndexResidentRatio': 0.07323550432920456, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 69.0771255493164, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 10813650399L, u'BloomFilterMemUsedRatio': 0.1326437145471573, u'NWriteBytes': 75998794927L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.10100603848695755, u'WriteAmp': 2.3519251346588135, u'NBlocksCached': 20130, u'LogicalDataSize': 24204242421L, u'ActiveBloomFilterMemUsed': 19365208, u'NFileCountCompacts': 5638, u'NTableFiles': 1421, u'FlushQueueSize': 72, u'TableObjectMemUsed': 366048, u'BlockCacheQuota': 102697451, u'TotalDiskUsage': 28609601536L, u'NPurgerRuns': 33, u'ReadIOAmpSet': 0.0, u'NSyncs': 73764, u'NReadIOsSetPerSec': 3.7666666507720947, u'NReadIOsPerSec': 241.23333740234375, u'NTablesDeleted': 103948, u'BloomFilterFPR': 0.01797540672123432, u'NBloomFilterHitsPerSec': 2672.833251953125, u'WriteCacheMemUsed': 3664968, u'RecentBloomFilterFPR': 0.021035796031355858, u'BytesPerRead': 953.9774169921875, u'BytesIncomingPerSec': 341674.71875, u'NReadIOsGetPerSec': 8.366666793823242, u'NGetStats': 144202, u'TotalIndexBlocksSize': 178909013, u'NGetStatsPerSec': 6.833333492279053, u'NWriteBytesCompact': 11519127552L, u'NReadIOsSet': 528471, u'ActiveDataSize': 24204242421L, u'NWriteBytesCompactPerSec': 611123.1875, u'NBloomFilterMissesPerSec': 57.43333435058594, u'NCommitBatches': 2712378, u'NGetStatsComputed': 144202, u'NWriteIOs': 2987318, u'NReadIOsGet': 3747409, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25674364, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 181.86666870117188, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 141696, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 66875216, u'NWriterCompacts': 0, u'NReadIOs': 8158288, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 273761933, u'NTablesCreated': 54292, u'ReadIOAmp': 0.12022897475431481, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 290461937, u'TotalIndexBlocksSize': 12594731, u'CheckpointOverhead': 27602944, u'NItemsGet': 31168851, u'NWriteIOs': 98677, u'NReadBytes': 22500388088L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2814188, u'SizeRatio': 0.06099588220769709, u'NWriteBytes': 1734356992, u'BloomFilterMemUsed': 40408, u'NumTables': 100, u'WastedSpace': 288, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2682880}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 29232038, u'SizeRatio': 0.6335873603820801, u'NWriteBytes': 3402903552L, u'BloomFilterMemUsed': 892800, u'NumTables': 56, u'WastedSpace': 53459, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 29233152}, {u'TargetSize': 79649470, u'LogicalSize': 49645549, u'SizeRatio': 0.6233004312520849, u'NWriteBytes': 4899434496L, u'BloomFilterMemUsed': 1533184, u'NumTables': 44, u'WastedSpace': 132350, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 49745920}, {u'TargetSize': 796494743, u'LogicalSize': 796494743, u'SizeRatio': 1.0, u'NWriteBytes': 3315761152L, u'BloomFilterMemUsed': 16897312, u'NumTables': 335, u'WastedSpace': 2358136, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 798842880}], u'NItemsGarbageCollected': 0, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 12011509, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 62496, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 8057896, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 197849, u'CompactWriter': {u'NDataBlockWrites': 4174697, u'NWriteBytes': 11496280064L, u'NIndexBlockWrites': 56004, u'NBytesIncoming': 15878137364L, u'NSyncs': 0, u'NInserts': 273761933, u'NWriteIOs': 26576}, u'NWriteBytes': 13264543114L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 3747399, u'NReadBytes': 10813646284L, u'NDataBlockReads': 24275712, u'NBytesOutgoing': 0, u'NIndexBlockReads': 35031975}, u'NBloomFilterMisses': 5316742, u'NItemsMerged': 0, u'LogicalDiskSize': 796494743, u'TotalBloomFilterMemUsed': 19779560, u'WriteAmp': 12.634241043143252, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 19363704, u'LogicalDataSize': 746849194, u'NTableFiles': 651, u'NFileCountCompacts': 2849, u'TableObjectMemUsed': 161448, u'TreeSnapshotMemoryUsed': 103088, u'TotalDiskUsage': 924389376, u'BytesIncoming': 1049888400, u'NItemsDel': 0, u'PendingCheckpointBytes': 32231424, u'NCompacts': 3400, u'NItemsPut': 21426350, u'NSyncs': 24588, u'CompactReader': {u'NReadIOs': 4310497, u'NReadBytes': 11686741804L, u'NDataBlockReads': 4211301, u'NBytesOutgoing': 0, u'NIndexBlockReads': 99196}, u'ActiveDiskUsage': 880504832, u'NTablesDeleted': 53757, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.01797540653699383, u'Writer': {u'NDataBlockWrites': 361987, u'NWriteBytes': 1768263050, u'NIndexBlockWrites': 47513, u'NBytesIncoming': 1242630662, u'NSyncs': 24588, u'NInserts': 21424714, u'NWriteIOs': 72101}}, u'NReadBytesCompactPerSec': 605857.75, u'TableMetaMemUsed': 431563, u'NDeletes': 0, u'BlockCacheHitsPerSec': 131.76666259765625, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 5316742, u'LogicalDiskSize': 24253887970L, u'NSets': 21426350, u'TotalBloomFilterMemUsed': 19781264, u'BufferMemUsed': 482326, u'RecentReadAmp': 0.0, u'NGetsPerSec': 82.83333587646484, u'NInserts': 0, u'NCommitBatchesPerSec': 152.06666564941406, u'FragmentationWithOverhead': 0.15398183465003967, u'NCompactsPerSec': 0.20000000298023224, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 629029.0625, u'TreeSnapshotMemoryUsed': 249440, u'BytesIncoming': 32313443320L, u'NCompacts': 6189, u'NumPersistentFlushes': 24588, u'NWriteIOsPerSec': 157.1666717529297, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 279.7344055175781, u'ActiveDiskUsage': 24338681856L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 6.833333492279053, u'NReadBytesSetPerSec': 10002.6337890625, u'BlockCacheHits': 12007990, u'NWriteBytesPerSec': 1510559.5, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 50302, u'ReadIOAmp': 0.0013605442176870747, u'MemtablesMemUsed': 959512, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 10524, u'CheckpointOverhead': 704512, u'NItemsGet': 7350, u'NWriteIOs': 74890, u'NReadBytes': 48803262, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 905976, u'SizeRatio': 0.0196365009654652, u'NWriteBytes': 389271552, u'BloomFilterMemUsed': 800, u'NumTables': 100, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 819200}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125515, u'SizeRatio': 0.002720464359630238, u'NWriteBytes': 22847488, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 15687003, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5460, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 20640, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 100392, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 61694, u'CompactWriter': {u'NDataBlockWrites': 2789, u'NWriteBytes': 22847488, u'NIndexBlockWrites': 2789, u'NBytesIncoming': 12332497, u'NSyncs': 0, u'NInserts': 147817, u'NWriteIOs': 2789}, u'NWriteBytes': 424509764, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 10, u'NReadBytes': 4115, u'NDataBlockReads': 6, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125515, u'TotalBloomFilterMemUsed': 1704, u'WriteAmp': 0.2662187847154323, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1504, u'LogicalDataSize': 125515, u'NTableFiles': 215, u'NFileCountCompacts': 2789, u'TableObjectMemUsed': 53320, u'TreeSnapshotMemoryUsed': 45552, u'TotalDiskUsage': 1761280, u'BytesIncoming': 1594589820, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 2789, u'NItemsPut': 15690614, u'NSyncs': 24588, u'CompactReader': {u'NReadIOs': 100382, u'NReadBytes': 48799147, u'NDataBlockReads': 50191, u'NBytesOutgoing': 0, u'NIndexBlockReads': 50191}, u'ActiveDiskUsage': 909312, u'NTablesDeleted': 50191, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 47513, u'NWriteBytes': 401662276, u'NIndexBlockWrites': 47513, u'NBytesIncoming': 34644652, u'NSyncs': 24588, u'NInserts': 447595, u'NWriteIOs': 72101}}}}} 2021-12-14 23:51:01,956 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.0020469108130782843, 0.0027165100909769535, 0.00139305938500911, 0.0019299933919683099, 0.000958361488301307, 0.002047860063612461, 0.0015450079226866364, 0.002323310123756528, 0.0018520269077271223, 0.0014646212803199887, 0.002221201779320836, 0.0019370371010154486, 0.0017453632317483425, 0.0025460789911448956, 0.0015319734811782837, 0.0021498773712664843, 0.0018240645295009017, 0.0025941880885511637, 0.002252208534628153, 0.0029648079071193933, 0.0026272947434335947, 0.002214686246588826, 0.0028201332315802574, 0.0018406849121674895, 0.002112727612257004, 0.002027262467890978, 0.0026600665878504515, 0.0018583416240289807, 0.0023526812437921762, 0.0025341715663671494, 0.0014149040216580033, 0.0018973700935021043, 0.0025034099817276, 0.0024971242528408766, 0.0022661639377474785, 0.0021393203642219305, 0.0015555989230051637, 0.0020580790005624294, 0.0022541119251400232, 0.002381180878728628, 0.002231736434623599, 0.0021957557182759047, 0.0017815438332036138, 0.002229947131127119, 0.002307100221514702, 0.0018497623968869448, 0.0018403661670163274, 0.0022485083900392056, 0.002746215555816889, 0.0019583581015467644, 0.0021051333751529455, 0.002193911001086235, 0.0018249292625114322, 0.0015995684079825878, 0.0015129817184060812, 0.001983410445973277, 0.0023125060833990574, 0.0024426982272416353, 0.002273965859785676, 0.0016614021733403206, 0.0022124657407402992, 0.0023821594659239054, 0.0023808612022548914, 0.0017620871076360345], '172.23.110.66': [0.002272434765473008, 0.0025508280377835035, 0.00221939361654222, 0.0020189255010336637, 0.0018907327903434634, 0.0021355259232223034, 0.0026536129880696535, 0.002164901699870825, 0.0022182504180818796, 0.0021764314733445644, 0.002375791547819972, 0.0021034160163253546, 0.0017078918172046542, 0.0019149143481627107, 0.001792319817468524, 0.00254348898306489, 0.0018039833521470428, 0.002473935717716813, 0.0017485232092440128, 0.0023192085791379213, 0.0020927302539348602, 0.0017937296070158482, 0.0026312104891985655, 0.0025638563092797995, 0.0021631489507853985, 0.002577486913651228, 0.0021224087104201317, 0.0028611321467906237, 0.0017839076463133097, 0.0023788237012922764, 0.002406564773991704, 0.002221534727141261, 0.002574535785242915, 0.002136275405064225, 0.002425646409392357, 0.0023432259913533926, 0.0023649309296160936, 0.0020661891903728247, 0.002200467512011528, 0.0027356999926269054, 0.0019484091317281127, 0.002441032789647579, 0.0023372406139969826, 0.0018935614498332143, 0.002085232175886631, 0.0017625887412577868, 0.0022905932273715734, 0.0018027309561148286, 0.0012595386942848563, 0.002315135207027197, 0.0024901465512812138, 0.0022186455316841602, 0.0025735951494425535, 0.0016933567821979523, 0.0027771175373345613, 0.0022027879022061825, 0.0017743963981047273, 0.001990478252992034, 0.0026365991216152906, 0.0021178401075303555, 0.002092172624543309, 0.002218046225607395, 0.0020724344067275524, 0.0022544877137988806], '172.23.110.64': [0.002263616770505905, 0.00229527335613966, 0.0019630894530564547, 0.0019174878252670169, 0.002764428034424782, 0.0020964043214917183, 0.0021523539908230305, 0.0019461410120129585, 0.0018254410242661834, 0.0019808041397482157, 0.0024313998874276876, 0.0019296554382890463, 0.0019361668964847922, 0.0013270671479403973, 0.002377398544922471, 0.0023250349331647158, 0.0023271474055945873, 0.0019191268365830183, 0.001939689740538597, 0.0019161192467436194, 0.0014497601659968495, 0.0021051259245723486, 0.0015244390815496445, 0.002285132883116603, 0.0021836368832737207, 0.0022848090156912804, 0.0017611836083233356, 0.0014204377075657248, 0.0016295555979013443, 0.002166987396776676, 0.0024394646752625704, 0.002391132293269038, 0.0024383682757616043, 0.0019332877127453685, 0.001463587861508131, 0.0021375033538788557, 0.0024780272506177425, 0.002279105130583048, 0.002397485775873065, 0.001954392297193408, 0.0018620690098032355, 0.0024596175644546747, 0.0018658506451174617, 0.002190160099416971, 0.0023047439754009247, 0.0019119448261335492, 0.002075818832963705, 0.0021120996680110693, 0.0024661305360496044, 0.0026045723352581263, 0.0017370876157656312, 0.0022990121506154537, 0.0023611581418663263, 0.0023965039290487766, 0.0019489629194140434, 0.0016252108616754413, 0.0020565122831612825, 0.002451986772939563, 0.002501843264326453, 0.002776282373815775, 0.0019243847345933318, 0.002088010311126709, 0.0020555504597723484, 0.002191368257626891]} 2021-12-14 23:51:02,325 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 1.0 2021-12-14 23:51:02,701 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 1.0 2021-12-14 23:51:03,055 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 1.0 2021-12-14 23:51:03,056 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 1.0, '172.23.110.66': 1.0, '172.23.110.64': 1.0} 2021-12-14 23:51:03,059 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-14 23:51:03,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:51:03,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-14 23:51:03,516 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:51:03,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-14 23:51:03,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-14 23:51:03,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-14 23:51:03,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-14 23:51:03,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-14 23:51:03,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-14 23:51:03,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-14 23:51:03,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-14 23:51:03,678 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-14 23:51:03,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-14 23:51:03,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:03,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-14 23:51:03,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:03,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:03,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-14 23:51:03,960 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-14 23:51:03,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-14 23:51:03,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:04,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-14 23:51:04,025 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-14 23:51:04,025 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-14 23:51:04,029 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-14 23:51:04,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:51:04,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-14 23:51:04,555 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:51:04,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-14 23:51:04,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-14 23:51:04,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-14 23:51:04,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-14 23:51:04,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-14 23:51:04,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-14 23:51:04,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-14 23:51:04,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-14 23:51:04,750 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-14 23:51:04,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-14 23:51:04,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:04,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-14 23:51:04,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:04,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:04,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-14 23:51:04,976 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-14 23:51:04,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-14 23:51:04,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:05,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-14 23:51:05,039 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-14 23:51:05,039 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-14 23:51:05,045 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-14 23:51:05,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-14 23:51:05,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-14 23:51:05,569 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-14 23:51:05,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-14 23:51:05,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-14 23:51:05,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-14 23:51:05,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-14 23:51:05,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-14 23:51:05,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-14 23:51:05,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-14 23:51:05,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-14 23:51:05,701 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-14 23:51:05,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-14 23:51:05,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:05,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-14 23:51:05,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:05,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-14 23:51:05,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-14 23:51:05,907 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-14 23:51:05,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-14 23:51:05,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-14 23:51:05,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-14 23:51:05,963 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-14 23:51:05,963 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 2.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6414.25390625 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 40000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 0 Final End: 40000000 2021-12-14 23:51:06,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:06,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1639554667.04 2021-12-14 23:51:07,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:07,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1639554668.06 2021-12-14 23:51:08,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:08,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1639554669.07 2021-12-14 23:51:09,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:09,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1639554670.08 2021-12-14 23:51:10,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:10,096 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1639554671.1 2021-12-14 23:51:11,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:11,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1639554672.11 2021-12-14 23:51:12,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:12,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1639554673.13 2021-12-14 23:51:13,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:13,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1639554674.21 2021-12-14 23:51:14,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:14,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1639554675.22 2021-12-14 23:51:15,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:15,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1639554676.23 2021-12-14 23:51:16,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:16,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:17,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:17,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:18,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:18,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:19,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:19,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:20,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:20,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:21,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:21,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:22,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:22,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:23,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:23,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:24,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:24,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:25,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:25,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:26,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:26,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:27,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:27,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:28,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:28,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:29,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:29,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:30,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:30,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:31,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:31,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:32,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:32,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:33,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:33,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:34,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:34,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:35,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:35,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:36,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:36,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:37,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:37,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:38,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:38,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:39,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:39,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:40,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:40,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:41,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:41,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:42,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:42,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:43,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:43,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:44,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:44,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:45,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:45,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:46,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:46,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:47,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:47,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:49,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:49,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:50,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:50,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:51,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:51,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:52,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:52,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:53,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:53,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:54,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:54,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:55,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:55,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-14 23:51:56,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-14 23:51:56,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection47_9_1639554668.06 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1639554677.25 Loader_GleamBookUsers0__default_VolumeCollection48_6_1639554671.1 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639554678.26 Loader_GleamBookUsers0__default_VolumeCollection42_4_1639554673.13 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639554679.28 Loader_GleamBookUsers0__default_VolumeCollection44_8_1639554669.07 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639554680.29 Loader_GleamBookUsers0__default_VolumeCollection46_10_1639554667.04 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639554681.37 Loader_GleamBookUsers0__default_VolumeCollection40_2_1639554675.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639554682.39 Loader_GleamBookUsers0__default_VolumeCollection45_7_1639554670.08 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639554683.4 Loader_GleamBookUsers0__default_VolumeCollection49_5_1639554672.11 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639554684.41 Loader_GleamBookUsers0__default_VolumeCollection41_1_1639554676.23 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639554685.43 Loader_GleamBookUsers0__default_VolumeCollection43_3_1639554674.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639554686.46 Loader_GleamBookUsers0__default_VolumeCollection35_0_1639554677.25 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639554687.61 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639554679.28 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639554688.62 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639554678.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639554689.63 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639554680.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639554690.65 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639554681.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639554691.66 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639554683.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639554692.68 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639554685.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639554693.69 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639554682.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639554694.7 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639554684.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639554695.72 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639554686.46 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639554696.73 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639554687.61 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639554697.74 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639554688.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639554698.75 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639554689.63 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639554699.78 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639554690.65 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639554700.8 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639554691.66 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639554701.81 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639554693.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639554702.84 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639554692.68 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639554703.85 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639554694.7 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639554704.86 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639554695.72 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639554705.88 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639554696.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639554706.89 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639554697.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639554707.9 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639554698.75 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639554709.03 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639554700.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639554710.04 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639554699.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639554711.06 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639554702.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639554712.07 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639554704.86 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639554713.08 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639554701.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639554714.1 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639554703.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639554715.11 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639554705.88 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639554716.12 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639554709.03 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639554717.14 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639554706.89 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639554711.06 is completed! Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639554707.9 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639554713.08 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639554710.04 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639554712.07 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639554714.1 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639554715.11 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639554716.12 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639554717.14 is completed! 2021-12-15 03:46:27,726 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-15 03:46:27,730 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 03:46:27,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:28,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 03:46:28,196 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:28,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 03:46:28,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 03:46:28,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 03:46:28,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 03:46:28,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 03:46:28,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 03:46:28,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 03:46:28,368 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 03:46:28,368 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568788.37 2021-12-15 03:46:28,371 | infra | INFO | pool-3-thread-11 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568788.37' started 2021-12-15 03:46:28,374 | infra | DEBUG | pool-3-thread-11 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 03:46:28,374 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 03:46:28,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:28,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 03:46:28,891 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:28,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 03:46:28,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 03:46:28,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 03:46:28,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 03:46:28,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 03:46:28,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 03:46:29,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 03:46:29,046 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 03:46:29,049 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.05 2021-12-15 03:46:29,052 | infra | INFO | pool-3-thread-12 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.05' started 2021-12-15 03:46:29,053 | infra | DEBUG | pool-3-thread-12 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 03:46:29,055 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 03:46:29,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:29,246 | test | DEBUG | pool-3-thread-11 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 03:46:29,247 | test | DEBUG | pool-3-thread-11 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-15 03:46:29,247 | infra | INFO | pool-3-thread-11 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568788.37' completed 2021-12-15 03:46:29,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 03:46:29,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:29,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 03:46:29,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 03:46:29,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 03:46:29,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 03:46:29,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 03:46:29,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 03:46:29,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 03:46:29,713 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 03:46:29,713 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.71 2021-12-15 03:46:29,714 | infra | INFO | pool-3-thread-13 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.71' started 2021-12-15 03:46:29,714 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568788.37 2021-12-15 03:46:29,716 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 03:46:29,716 | infra | DEBUG | pool-3-thread-13 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 03:46:29,716 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.05 2021-12-15 03:46:29,911 | test | DEBUG | pool-3-thread-12 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 03:46:29,913 | test | DEBUG | pool-3-thread-12 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-15 03:46:29,913 | infra | INFO | pool-3-thread-12 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.05' completed 2021-12-15 03:46:29,914 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 03:46:29,914 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.71 2021-12-15 03:46:30,510 | test | DEBUG | pool-3-thread-13 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 03:46:30,510 | test | DEBUG | pool-3-thread-13 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-15 03:46:30,512 | infra | INFO | pool-3-thread-13 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639568789.71' completed 2021-12-15 03:46:30,512 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 03:46:30,555 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 20013.4, u'diskFetches': 0, u'dataUsed': 3779133111566L, u'itemCount': 2000000000, u'quotaPercentUsed': 78.72220002363126, u'storageTotals': {u'hdd': {u'usedByData': 6773959420070L, u'used': 6940165511574L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 29035644479082L}, u'ram': {u'usedByData': 253581955928L, u'used': 361602916352L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 6773959420070L, u'vbActiveNumNonResident': 1864203941, u'memUsed': 253581955928L} 2021-12-15 03:46:30,917 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 2000000000 | 300.00 GiB | 236.17 GiB | 6.16 TiB | 6.79443025 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ 2021-12-15 03:46:30,917 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+---------------+---------------+---------------+ | 2000000000 | 2000000000 | 0-40000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+---------------+---------------+---------------+---------------+ 2021-12-15 03:46:31,289 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 6.79443025 2021-12-15 03:46:31,292 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 03:46:31,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:31,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 03:46:31,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:31,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 03:46:31,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 03:46:31,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 03:46:31,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 03:46:31,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 03:46:31,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 03:46:31,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 03:46:31,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 03:46:32,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 03:46:32,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 03:46:32,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 03:46:32,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 03:46:34,691 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 03:46:34,696 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 03:46:34,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:35,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 03:46:35,203 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:35,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 03:46:35,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 03:46:35,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 03:46:35,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 03:46:35,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 03:46:35,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 03:46:35,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 03:46:35,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 03:46:35,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 03:46:35,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 03:46:35,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 03:46:36,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 03:46:38,006 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 03:46:38,010 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 03:46:38,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:38,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 03:46:38,559 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:38,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 03:46:38,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 03:46:38,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 03:46:38,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 03:46:38,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 03:46:38,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 03:46:38,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 03:46:38,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 03:46:39,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 03:46:39,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 03:46:39,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 03:46:39,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 03:46:41,292 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 03:46:41,292 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 6474870MB 2021-12-15 03:46:41,293 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 10607MB 2021-12-15 03:46:41,293 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 190529MB 2021-12-15 03:46:41,293 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 6283940MB 2021-12-15 03:46:41,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 03:46:41,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:46:41,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 03:46:41,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:46:41,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 03:46:41,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 03:46:41,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 03:46:41,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 03:46:41,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 03:46:41,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 03:46:41,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 03:46:41,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 03:46:42,048 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-15 03:46:42,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 03:46:42,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 03:46:42,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 03:46:43,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 03:46:43,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 03:46:44,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 03:46:44,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 03:46:45,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 03:46:45,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 03:46:46,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 03:46:46,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 03:46:47,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 03:46:47,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 03:46:48,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 03:46:48,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 03:46:49,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 03:46:49,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 03:46:49,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 03:46:50,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 03:46:50,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 03:46:51,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 03:46:51,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 03:46:51,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 03:46:52,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 03:46:52,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 03:46:53,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 03:46:53,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 03:46:54,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 03:46:54,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 03:46:54,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 03:46:55,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 03:46:55,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 03:46:56,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 03:46:56,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 03:46:57,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 03:46:57,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 03:46:58,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 03:46:58,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 03:46:59,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 03:46:59,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 03:46:59,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 03:47:00,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 03:47:00,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 03:47:00,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 03:47:01,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 03:47:01,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 03:47:02,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 03:47:02,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 03:47:02,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 03:47:03,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 03:47:03,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 03:47:04,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 03:47:04,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 03:47:05,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 03:47:05,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 03:47:05,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 03:47:06,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 03:47:06,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 03:47:07,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 03:47:07,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 03:47:08,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 03:47:08,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 03:47:09,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 03:47:09,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 03:47:09,960 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 03:47:09,963 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 03:47:10,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:47:10,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 03:47:10,505 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:47:10,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 03:47:10,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 03:47:10,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 03:47:10,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 03:47:10,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 03:47:10,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 03:47:10,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 03:47:10,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 03:47:10,727 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-15 03:47:10,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 03:47:11,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 03:47:11,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 03:47:12,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 03:47:12,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 03:47:13,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 03:47:13,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 03:47:13,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 03:47:14,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 03:47:14,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 03:47:15,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 03:47:15,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 03:47:16,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 03:47:16,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 03:47:17,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 03:47:17,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 03:47:18,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 03:47:18,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 03:47:19,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 03:47:19,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 03:47:19,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 03:47:20,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 03:47:20,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 03:47:21,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 03:47:21,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 03:47:22,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 03:47:22,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 03:47:22,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 03:47:23,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 03:47:23,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 03:47:24,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 03:47:24,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 03:47:25,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 03:47:25,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 03:47:26,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 03:47:26,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 03:47:26,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 03:47:27,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 03:47:27,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 03:47:28,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 03:47:28,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 03:47:29,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 03:47:29,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 03:47:30,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 03:47:30,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 03:47:31,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 03:47:31,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 03:47:32,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 03:47:32,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 03:47:32,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 03:47:33,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 03:47:33,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 03:47:34,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 03:47:34,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 03:47:35,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 03:47:35,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 03:47:36,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 03:47:36,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 03:47:36,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 03:47:37,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 03:47:37,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 03:47:38,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 03:47:38,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 03:47:39,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 03:47:39,811 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 03:47:39,813 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 03:47:39,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:47:40,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 03:47:40,270 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:47:40,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 03:47:40,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 03:47:40,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 03:47:40,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 03:47:40,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 03:47:40,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 03:47:40,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 03:47:40,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 03:47:40,486 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-15 03:47:40,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 03:47:40,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 03:47:41,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 03:47:41,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 03:47:42,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 03:47:42,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 03:47:43,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 03:47:43,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 03:47:44,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 03:47:44,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 03:47:45,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 03:47:45,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 03:47:46,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 03:47:46,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 03:47:47,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 03:47:47,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 03:47:48,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 03:47:48,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 03:47:49,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 03:47:49,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 03:47:50,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 03:47:50,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 03:47:51,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 03:47:51,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 03:47:52,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 03:47:52,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 03:47:53,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 03:47:53,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 03:47:53,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 03:47:54,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 03:47:54,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 03:47:55,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 03:47:55,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 03:47:56,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 03:47:56,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 03:47:57,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 03:47:57,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 03:47:58,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 03:47:58,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 03:47:59,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 03:47:59,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 03:47:59,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 03:48:00,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 03:48:00,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 03:48:01,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 03:48:01,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 03:48:02,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 03:48:02,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 03:48:03,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 03:48:03,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 03:48:03,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 03:48:04,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 03:48:04,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 03:48:04,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 03:48:05,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 03:48:05,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 03:48:06,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 03:48:06,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 03:48:07,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 03:48:07,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 03:48:08,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 03:48:08,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 03:48:09,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 03:48:09,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 03:48:09,983 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 03:48:09,983 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 30401.833984375, u'ReplayWALOffset': u'5240:1949696', u'NTablesCreated': 401484, u'RecentReadIOAmpSet': 0.08513154089450836, u'ReadIOAmp': 613.8389282226562, u'BlockCacheMisses': 24000138, u'NFlushes': 90611, u'RecentWriteAmp': 1.898686170578003, u'NumSyncFlushes': 14, u'TotalMemUsed': 124170277, u'NBloomFilterHits': 482543955, u'Fragmentation': 0.4386647641658783, u'NumNormalFlushes': 47588, u'RecentBlockCacheHitRatio': 97.3801498413086, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 213618, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 7070744, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 279129885, u'CheckpointOverhead': 2863104, u'NItemsGet': 0, u'NWriteIOs': 316051, u'NReadBytes': 15594929163L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 26719637, u'SizeRatio': 0.5791325352408669, u'NWriteBytes': 720785408, u'BloomFilterMemUsed': 0, u'NumTables': 75, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 962560}, {u'TargetSize': 46137344, u'LogicalSize': 25555377, u'SizeRatio': 0.5538978793404319, u'NWriteBytes': 580300800, u'BloomFilterMemUsed': 0, u'NumTables': 21, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 364544}, {u'TargetSize': 344483719, u'LogicalSize': 335025746, u'SizeRatio': 0.9725444992655807, u'NWriteBytes': 1554919424, u'BloomFilterMemUsed': 0, u'NumTables': 177, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 4091904}, {u'TargetSize': 17570500608L, u'LogicalSize': 15726062938L, u'SizeRatio': 0.8950264587703203, u'NWriteBytes': 1920401408, u'BloomFilterMemUsed': 0, u'NumTables': 486, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 154808320}, {u'TargetSize': 68896744972L, u'LogicalSize': 35141001216L, u'SizeRatio': 0.5100531415567091, u'NWriteBytes': 47238033408L, u'BloomFilterMemUsed': 0, u'NumTables': 181, u'WastedSpace': 692628730, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 35141001216L}], u'NItemsGarbageCollected': 10695761, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 213518622, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 111072, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5231726, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 326274, u'CompactWriter': {u'NDataBlockWrites': 1421790, u'NWriteBytes': 4318363648L, u'NIndexBlockWrites': 96369, u'NBytesIncoming': 5782284366L, u'NSyncs': 0, u'NInserts': 238012432, u'NWriteIOs': 92890}, u'NWriteBytes': 51975869393L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 10695761, u'LogicalDiskSize': 35141001216L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8352165386512039, u'BufferMemUsed': 498818, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 19414938278L, u'NTableFiles': 1102, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 286936, u'TreeSnapshotMemoryUsed': 136576, u'TotalDiskUsage': 46065491968L, u'BytesIncoming': 62230412100L, u'NItemsDel': 0, u'PendingCheckpointBytes': 17903616, u'NCompacts': 30067, u'NItemsPut': 64277950, u'NSyncs': 43023, u'CompactReader': {u'NReadIOs': 5231726, u'NReadBytes': 15594929163L, u'NDataBlockReads': 5073634, u'NBytesOutgoing': 0, u'NIndexBlockReads': 158092}, u'ActiveDiskUsage': 35301228544L, u'NTablesDeleted': 128811, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 14447215, u'NWriteBytes': 47657505745L, u'NIndexBlockWrites': 353190, u'NBytesIncoming': 62802612546L, u'NSyncs': 43023, u'NInserts': 64269416, u'NWriteIOs': 223161}}, u'NReadBytes': 16085737763465L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 9.600000381469727, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 226913529, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5052, u'NTxns': 3997976, u'NWriteBytes': 74339905536L, u'NSegments': 5243, u'NSyncs': 3997976, u'NLoggedBytes': 74339905536L, u'NActiveSegments': 4, u'NWrites': 4022032, u'NInserts': 79586598, u'DiskUsed': 56672256}, u'NGets': 9742501, u'NReadBytesSet': 34482398965L, u'NReadBytesCompact': 43239589731L, u'IndexResidentRatio': 0.0838114395737648, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.0, u'BlockCacheHitRatio': 74.04802703857422, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 16042498173734L, u'BloomFilterMemUsedRatio': 0.14322659373283386, u'NWriteBytes': 157016566498L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.4600656032562256, u'NBlocksCached': 28013, u'LogicalDataSize': 20215766466L, u'ActiveBloomFilterMemUsed': 20882464, u'NFileCountCompacts': 9981, u'NTableFiles': 1862, u'FlushQueueSize': 72, u'TableObjectMemUsed': 475416, u'BlockCacheQuota': 101323976, u'TotalDiskUsage': 47093702656L, u'NPurgerRuns': 77, u'ReadIOAmpSet': 0.0, u'NSyncs': 129069, u'NReadIOsSetPerSec': 9.600000381469727, u'NReadIOsPerSec': 11.600000381469727, u'NTablesDeleted': 316150, u'BloomFilterFPR': 0.015222710557281971, u'NBloomFilterHitsPerSec': 819.7666625976562, u'WriteCacheMemUsed': 9825352, u'RecentBloomFilterFPR': 0.016083216294646263, u'BytesPerRead': 1643111.625, u'BytesIncomingPerSec': 212707.9375, u'NReadIOsGetPerSec': 11.600000381469727, u'NGetStats': 221669, u'TotalIndexBlocksSize': 293162838, u'NGetStatsPerSec': 4.0333333015441895, u'NWriteBytesCompact': 30855012352L, u'NReadIOsSet': 13006138, u'ActiveDataSize': 20215766466L, u'NWriteBytesCompactPerSec': 0.0, u'NBloomFilterMissesPerSec': 13.399999618530273, u'NCommitBatches': 3997976, u'NGetStatsComputed': 221669, u'NWriteIOs': 4657845, u'NReadIOsGet': 5993332338L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 25330994, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 112.76667022705078, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.0, u'ManagedFileObjectMemUsed': 184032, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 90502320, u'NWriterCompacts': 0, u'NReadIOs': 6008963735L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 634373141, u'NTablesCreated': 98395, u'ReadIOAmp': 113.95326503830209, u'MemtablesMemUsed': 435224, u'NBloomFilterHits': 482543955, u'TotalIndexBlocksSize': 14021685, u'CheckpointOverhead': 25534464, u'NItemsGet': 52594651, u'NWriteIOs': 187268, u'NReadBytes': 16070054123738L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 2345304, u'SizeRatio': 0.05083309520374645, u'NWriteBytes': 3387088896L, u'BloomFilterMemUsed': 32488, u'NumTables': 88, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 2228224}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 29846532, u'SizeRatio': 0.6469061591408469, u'NWriteBytes': 10554130432L, u'BloomFilterMemUsed': 918656, u'NumTables': 59, u'WastedSpace': 54527, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 29831168}, {u'TargetSize': 87258638, u'LogicalSize': 71883686, u'SizeRatio': 0.8238002293824481, u'NWriteBytes': 7313616896L, u'BloomFilterMemUsed': 2202624, u'NumTables': 44, u'WastedSpace': 187348, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 72056832}, {u'TargetSize': 872586436, u'LogicalSize': 872586436, u'SizeRatio': 1.0, u'NWriteBytes': 8913797120L, u'BloomFilterMemUsed': 17727328, u'NumTables': 242, u'WastedSpace': 2803958, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 875364352}], u'NItemsGarbageCollected': 18930383, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 13390263, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 50880, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6003553122L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 161065, u'CompactWriter': {u'NDataBlockWrites': 9663166, u'NWriteBytes': 26495967232L, u'NIndexBlockWrites': 126657, u'NBytesIncoming': 36793576928L, u'NSyncs': 0, u'NInserts': 634373141, u'NWriteIOs': 59649}, u'NWriteBytes': 29945976326L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5993332205L, u'NReadBytes': 16042498104399L, u'NDataBlockReads': 13213405343L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 25697696496L}, u'NBloomFilterMisses': 7459176, u'NItemsMerged': 0, u'LogicalDiskSize': 872586436, u'TotalBloomFilterMemUsed': 21358128, u'WriteAmp': 14.261668887086984, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 20881096, u'LogicalDataSize': 800702750, u'NTableFiles': 530, u'NFileCountCompacts': 5015, u'TableObjectMemUsed': 131440, u'TreeSnapshotMemoryUsed': 92624, u'TotalDiskUsage': 1026326528, u'BytesIncoming': 2099752600, u'NItemsDel': 0, u'PendingCheckpointBytes': 49242112, u'NCompacts': 6465, u'NItemsPut': 42852150, u'NSyncs': 43023, u'CompactReader': {u'NReadIOs': 10220917, u'NReadBytes': 27556019339L, u'NDataBlockReads': 10014006, u'NBytesOutgoing': 0, u'NIndexBlockReads': 206911}, u'ActiveDiskUsage': 979480576, u'NTablesDeleted': 97962, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015222710893249374, u'Writer': {u'NDataBlockWrites': 713976, u'NWriteBytes': 3450009094L, u'NIndexBlockWrites': 84791, u'NBytesIncoming': 2485174464L, u'NSyncs': 43023, u'NInserts': 42847883, u'NWriteIOs': 127619}}, u'NReadBytesCompactPerSec': 0.0, u'TableMetaMemUsed': 553347, u'NDeletes': 0, u'BlockCacheHitsPerSec': 356.8333435058594, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 7459176, u'LogicalDiskSize': 36013713090L, u'NSets': 42852150, u'TotalBloomFilterMemUsed': 21359496, u'BufferMemUsed': 498818, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 95.80000305175781, u'FragmentationWithOverhead': 0.5707331299781799, u'NCompactsPerSec': 0.0, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 30401.833984375, u'TreeSnapshotMemoryUsed': 274304, u'BytesIncoming': 63826168465L, u'NCompacts': 41498, u'NumPersistentFlushes': 43009, u'NWriteIOsPerSec': 95.80000305175781, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 36281479168L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 4.0333333015441895, u'NReadBytesSetPerSec': 25350.099609375, u'BlockCacheHits': 68478920, u'NWriteBytesPerSec': 403865.59375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 89471, u'ReadIOAmp': 0.00986646884272997, u'MemtablesMemUsed': 2319384, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 11268, u'CheckpointOverhead': 1114112, u'NItemsGet': 13480, u'NWriteIOs': 132494, u'NReadBytes': 88710564, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 753229, u'SizeRatio': 0.01632579890164462, u'NWriteBytes': 692310016, u'BloomFilterMemUsed': 664, u'NumTables': 83, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 679936}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125438, u'SizeRatio': 0.002718795429576527, u'NWriteBytes': 40681472, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 24732517, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 4644, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 22080, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 178887, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 66008, u'CompactWriter': {u'NDataBlockWrites': 4966, u'NWriteBytes': 40681472, u'NIndexBlockWrites': 4966, u'NBytesIncoming': 22310210, u'NSyncs': 0, u'NInserts': 263198, u'NWriteIOs': 4966}, u'NWriteBytes': 754815243, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 133, u'NReadBytes': 69335, u'NDataBlockReads': 114, u'NBytesOutgoing': 0, u'NIndexBlockReads': 114}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125438, u'TotalBloomFilterMemUsed': 1368, u'WriteAmp': 0.31601829271035736, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1368, u'LogicalDataSize': 125438, u'NTableFiles': 230, u'NFileCountCompacts': 4966, u'TableObjectMemUsed': 57040, u'TreeSnapshotMemoryUsed': 45104, u'TotalDiskUsage': 1884160, u'BytesIncoming': 2388517565L, u'NItemsDel': 0, u'PendingCheckpointBytes': 16384, u'NCompacts': 4966, u'NItemsPut': 24740534, u'NSyncs': 43023, u'CompactReader': {u'NReadIOs': 178754, u'NReadBytes': 88641229, u'NDataBlockReads': 89377, u'NBytesOutgoing': 0, u'NIndexBlockReads': 89377}, u'ActiveDiskUsage': 770048, u'NTablesDeleted': 89377, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 84505, u'NWriteBytes': 714133771, u'NIndexBlockWrites': 84505, u'NBytesIncoming': 63086585, u'NSyncs': 43023, u'NInserts': 834743, u'NWriteIOs': 127528}}}}} 2021-12-15 03:48:09,987 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.4386647641658783, 0.4360074996948242, 0.42890650033950806, 0.4431287348270416, 0.43617331981658936, 0.4401748776435852, 0.42707958817481995, 0.43307361006736755, 0.43901193141937256, 0.4356151819229126, 0.43477022647857666, 0.43674522638320923, 0.4283473789691925, 0.43417033553123474, 0.43588411808013916, 0.4368278980255127, 0.4338417649269104, 0.4361688494682312, 0.4410301446914673, 0.4322781562805176, 0.43822452425956726, 0.43976280093193054, 0.4368252158164978, 0.43314170837402344, 0.4361608326435089, 0.4346669614315033, 0.4395170509815216, 0.43187475204467773, 0.4345035254955292, 0.43330836296081543, 0.43682006001472473, 0.4361012876033783, 0.4377550184726715, 0.43239349126815796, 0.4398518204689026, 0.43651413917541504, 0.44128650426864624, 0.43981432914733887, 0.4378054738044739, 0.4345681071281433, 0.4352512061595917, 0.4298003613948822, 0.43028369545936584, 0.44130146503448486, 0.43089017271995544, 0.4393411874771118, 0.4312567412853241, 0.4346294701099396, 0.4313717484474182, 0.43796882033348083, 0.43672192096710205, 0.44110679626464844, 0.43805840611457825, 0.43771156668663025, 0.43936261534690857, 0.44174522161483765, 0.4372458755970001, 0.4366232752799988, 0.4348522424697876, 0.4373975694179535, 0.435046911239624, 0.44076117873191833, 0.43036308884620667, 0.42941635847091675], '172.23.110.66': [0.4374152421951294, 0.43779391050338745, 0.43968790769577026, 0.43899860978126526, 0.4338013231754303, 0.44510963559150696, 0.4391614496707916, 0.43231305480003357, 0.43443387746810913, 0.43520525097846985, 0.43467751145362854, 0.4423198997974396, 0.43568652868270874, 0.43434616923332214, 0.43919286131858826, 0.4364873766899109, 0.43159860372543335, 0.4414041340351105, 0.44202494621276855, 0.4399969279766083, 0.4362589716911316, 0.4369698166847229, 0.43877363204956055, 0.4388619661331177, 0.43225184082984924, 0.44159993529319763, 0.44050589203834534, 0.4362119734287262, 0.44179457426071167, 0.43537765741348267, 0.4425524175167084, 0.43561503291130066, 0.44091325998306274, 0.43934372067451477, 0.4333358108997345, 0.43831169605255127, 0.43698373436927795, 0.4398840665817261, 0.4375380873680115, 0.4341025650501251, 0.4327322840690613, 0.4321776330471039, 0.43859899044036865, 0.4402838945388794, 0.4363848865032196, 0.43763861060142517, 0.44340625405311584, 0.4443379044532776, 0.43280932307243347, 0.43935006856918335, 0.43092572689056396, 0.4392589032649994, 0.43803703784942627, 0.44013717770576477, 0.44129741191864014, 0.42865225672721863, 0.4366980195045471, 0.4383606016635895, 0.4348160922527313, 0.4401412010192871, 0.4306710958480835, 0.43495887517929077, 0.4290611743927002, 0.4358438551425934], '172.23.110.64': [0.43829792737960815, 0.4300423264503479, 0.433287650346756, 0.42878592014312744, 0.4371355473995209, 0.43610548973083496, 0.44117271900177, 0.44002196192741394, 0.4294275939464569, 0.434152752161026, 0.42719605565071106, 0.43766847252845764, 0.42622682452201843, 0.4343811869621277, 0.4373893737792969, 0.43763306736946106, 0.4295060634613037, 0.4325072169303894, 0.43169593811035156, 0.4273829460144043, 0.43787238001823425, 0.4305959939956665, 0.4332965612411499, 0.43876293301582336, 0.4324035942554474, 0.4369348883628845, 0.4429745078086853, 0.44475671648979187, 0.4333338141441345, 0.4333953559398651, 0.44036421179771423, 0.4401579201221466, 0.4293137788772583, 0.42933380603790283, 0.4446471035480499, 0.4284384846687317, 0.4244340658187866, 0.4368657171726227, 0.43044641613960266, 0.43868786096572876, 0.435190886259079, 0.44001317024230957, 0.4379497468471527, 0.4331819415092468, 0.43026360869407654, 0.436168372631073, 0.43522846698760986, 0.4385688304901123, 0.43594735860824585, 0.43337687849998474, 0.43769314885139465, 0.4256620705127716, 0.4300359785556793, 0.4301747679710388, 0.43722057342529297, 0.4287974238395691, 0.438059002161026, 0.43363213539123535, 0.44061580300331116, 0.436338871717453, 0.430951863527298, 0.4329178035259247, 0.4395602345466614, 0.4327433109283447]} 2021-12-15 03:48:10,344 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 44.0 2021-12-15 03:48:10,710 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 44.0 2021-12-15 03:48:11,028 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 44.0 2021-12-15 03:48:11,028 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 44.0, '172.23.110.66': 44.0, '172.23.110.64': 44.0} 2021-12-15 03:48:11,029 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 03:48:11,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:48:11,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 03:48:11,466 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:48:11,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 03:48:11,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 03:48:11,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 03:48:11,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 03:48:11,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 03:48:11,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 03:48:11,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 03:48:11,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 03:48:11,635 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 03:48:11,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 03:48:11,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:11,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 03:48:11,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:11,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:11,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:11,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 03:48:11,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 03:48:11,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 03:48:11,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 03:48:11,979 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 03:48:11,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 03:48:12,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:12,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 03:48:12,051 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 03:48:12,052 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 03:48:12,055 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 03:48:12,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:48:12,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 03:48:12,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:48:12,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 03:48:12,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 03:48:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 03:48:12,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 03:48:12,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 03:48:12,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 03:48:12,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 03:48:12,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 03:48:12,714 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 03:48:12,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 03:48:12,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:12,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 03:48:12,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:12,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:12,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 03:48:12,950 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 03:48:12,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 03:48:12,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:12,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 03:48:13,016 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 03:48:13,016 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 03:48:13,019 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 03:48:13,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 03:48:13,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 03:48:13,533 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 03:48:13,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 03:48:13,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 03:48:13,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 03:48:13,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 03:48:13,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 03:48:13,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 03:48:13,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 03:48:13,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 03:48:13,668 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 03:48:13,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 03:48:13,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:13,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 03:48:13,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:13,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 03:48:13,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 03:48:13,872 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 03:48:13,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 03:48:13,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 03:48:13,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 03:48:13,921 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 03:48:13,921 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3: Create 40000000 items sequentially # ############################################################ ############################################################ # # RAM FootPrint: 6862.83984375 # ############################################################ Read Start: 0 Read End: 0 Update Start: 0 Update End: 0 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 40000000 Create End: 80000000 Final Start: 40000000 Final End: 80000000 2021-12-15 03:48:13,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:13,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1639568894.98 2021-12-15 03:48:14,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:14,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1639568895.99 2021-12-15 03:48:15,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:15,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1639568897.0 2021-12-15 03:48:17,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:17,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1639568898.01 2021-12-15 03:48:18,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:18,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1639568899.02 2021-12-15 03:48:19,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:19,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1639568900.03 2021-12-15 03:48:20,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:20,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1639568901.05 2021-12-15 03:48:21,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:21,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1639568902.06 2021-12-15 03:48:22,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:22,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1639568903.07 2021-12-15 03:48:23,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:23,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1639568904.09 2021-12-15 03:48:24,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:24,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:25,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:25,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:26,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:26,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:27,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:27,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:28,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:28,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:29,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:29,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:30,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:30,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:31,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:31,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:32,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:32,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:33,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:33,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:34,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:34,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:35,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:35,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:36,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:36,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:37,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:37,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:38,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:38,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:39,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:39,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:40,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:40,313 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:41,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:41,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:42,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:42,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:43,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:43,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:44,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:44,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:45,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:45,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:46,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:46,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:47,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:47,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:48,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:48,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:49,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:49,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:50,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:50,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:51,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:51,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:52,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:52,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:53,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:53,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:54,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:54,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:55,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:55,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:56,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:56,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:57,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:57,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:58,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:58,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:48:59,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:48:59,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:49:00,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:49:00,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:49:01,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:49:01,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:49:02,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:49:02,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 03:49:03,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 03:49:03,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection44_8_1639568897.0 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1639568905.1 Loader_GleamBookUsers0__default_VolumeCollection42_4_1639568901.05 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639568906.11 Loader_GleamBookUsers0__default_VolumeCollection49_5_1639568900.03 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639568907.12 Loader_GleamBookUsers0__default_VolumeCollection47_9_1639568895.99 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639568908.14 Loader_GleamBookUsers0__default_VolumeCollection45_7_1639568898.01 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639568909.15 Loader_GleamBookUsers0__default_VolumeCollection43_3_1639568902.06 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639568910.16 Loader_GleamBookUsers0__default_VolumeCollection41_1_1639568904.09 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639568911.17 Loader_GleamBookUsers0__default_VolumeCollection46_10_1639568894.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639568912.19 Loader_GleamBookUsers0__default_VolumeCollection40_2_1639568903.07 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639568913.2 Loader_GleamBookUsers0__default_VolumeCollection48_6_1639568899.02 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639568914.21 Loader_GleamBookUsers0__default_VolumeCollection35_0_1639568905.1 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639568915.22 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639568909.15 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639568916.24 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639568906.11 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639568917.26 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639568911.17 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639568918.28 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639568908.14 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639568919.29 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639568910.16 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639568920.3 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639568913.2 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639568921.31 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639568907.12 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639568922.33 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639568914.21 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639568923.36 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639568912.19 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639568924.37 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639568915.22 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639568925.38 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639568917.26 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639568926.39 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639568919.29 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639568927.4 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639568921.31 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639568928.42 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639568916.24 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639568929.43 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639568923.36 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639568930.44 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639568918.28 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639568931.45 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639568920.3 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639568932.46 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639568922.33 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639568933.48 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639568924.37 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639568934.49 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639568925.38 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639568935.5 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639568927.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639568936.52 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639568929.43 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639568937.54 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639568931.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639568938.55 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639568926.39 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639568939.56 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639568928.42 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639568940.75 Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639568935.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639568941.76 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639568930.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639568942.77 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639568932.46 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639568943.78 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639568933.48 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639568944.8 Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639568939.56 is completed! Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639568936.52 is completed! Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639568934.49 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639568937.54 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639568941.76 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639568938.55 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639568940.75 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639568943.78 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639568942.77 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639568944.8 is completed! 2021-12-15 07:56:40,377 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-15 07:56:40,382 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 07:56:40,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:40,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 07:56:40,878 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:40,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 07:56:40,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 07:56:40,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 07:56:40,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 07:56:40,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 07:56:41,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 07:56:41,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 07:56:41,046 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 07:56:41,049 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.05 2021-12-15 07:56:41,049 | infra | INFO | pool-3-thread-14 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.05' started 2021-12-15 07:56:41,053 | infra | DEBUG | pool-3-thread-14 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 07:56:41,056 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 07:56:41,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:41,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 07:56:41,607 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:41,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 07:56:41,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 07:56:41,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 07:56:41,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 07:56:41,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 07:56:41,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 07:56:41,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 07:56:41,750 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 07:56:41,750 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.75 2021-12-15 07:56:41,752 | infra | INFO | pool-3-thread-15 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.75' started 2021-12-15 07:56:41,753 | infra | DEBUG | pool-3-thread-15 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 07:56:41,756 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 07:56:41,812 | test | DEBUG | pool-3-thread-14 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 07:56:41,812 | test | DEBUG | pool-3-thread-14 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-15 07:56:41,812 | infra | INFO | pool-3-thread-14 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.05' completed 2021-12-15 07:56:41,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:42,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 07:56:42,296 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:42,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 07:56:42,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 07:56:42,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 07:56:42,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 07:56:42,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 07:56:42,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 07:56:42,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 07:56:42,408 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 07:56:42,408 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583802.41 2021-12-15 07:56:42,410 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.05 2021-12-15 07:56:42,410 | infra | INFO | pool-3-thread-16 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583802.41' started 2021-12-15 07:56:42,411 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 07:56:42,411 | infra | DEBUG | pool-3-thread-16 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 07:56:42,411 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.75 2021-12-15 07:56:42,427 | test | DEBUG | pool-3-thread-15 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 07:56:42,427 | test | DEBUG | pool-3-thread-15 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-15 07:56:42,427 | infra | INFO | pool-3-thread-15 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583801.75' completed 2021-12-15 07:56:42,428 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 07:56:42,428 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583802.41 2021-12-15 07:56:43,250 | test | DEBUG | pool-3-thread-16 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 07:56:43,250 | test | DEBUG | pool-3-thread-16 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-15 07:56:43,250 | infra | INFO | pool-3-thread-16 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639583802.41' completed 2021-12-15 07:56:43,250 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 07:56:43,292 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 19430.8, u'diskFetches': 0, u'dataUsed': 8185828053573L, u'itemCount': 3999890270L, u'quotaPercentUsed': 77.65544968346755, u'storageTotals': {u'hdd': {u'usedByData': 11303447866689L, u'used': 11432856681180L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 24542953309476L}, u'ram': {u'usedByData': 250145712560L, u'used': 382660218880L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 11303447866689L, u'vbActiveNumNonResident': 3865829668L, u'memUsed': 250145712560L} 2021-12-15 07:56:43,658 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999890270 | 300.00 GiB | 232.97 GiB | 10.28 TiB | 3.35425695 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ 2021-12-15 07:56:43,658 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+---------------+-------------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+---------------+-------------------+---------------+---------------+ | 2000000000 | 4000000000 | 0-0 | 40000000-80000000 | 0-0 | 0-0 | +---------------+---------------+---------------+-------------------+---------------+---------------+ 2021-12-15 07:56:44,030 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.35425695 2021-12-15 07:56:44,032 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 07:56:44,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:44,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 07:56:44,490 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:44,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 07:56:44,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 07:56:44,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 07:56:44,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 07:56:44,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 07:56:44,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 07:56:44,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 07:56:44,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 07:56:45,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 07:56:45,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 07:56:45,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 07:56:45,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 07:56:47,299 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 07:56:47,305 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 07:56:47,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:47,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 07:56:47,816 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:47,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 07:56:47,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 07:56:47,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 07:56:47,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 07:56:47,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 07:56:47,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 07:56:47,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 07:56:47,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 07:56:48,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 07:56:48,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 07:56:48,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 07:56:48,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 07:56:50,463 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 07:56:50,467 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 07:56:50,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:50,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 07:56:51,010 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:51,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 07:56:51,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 07:56:51,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 07:56:51,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 07:56:51,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 07:56:51,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 07:56:51,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 07:56:51,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 07:56:51,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 07:56:51,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 07:56:51,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 07:56:52,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 07:56:53,698 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 07:56:53,700 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 10820779MB 2021-12-15 07:56:53,700 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 7941MB 2021-12-15 07:56:53,700 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 346646MB 2021-12-15 07:56:53,700 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 10473723MB 2021-12-15 07:56:53,703 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 07:56:53,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:56:54,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 07:56:54,211 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:56:54,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 07:56:54,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 07:56:54,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 07:56:54,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 07:56:54,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 07:56:54,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 07:56:54,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 07:56:54,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 07:56:54,371 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-15 07:56:54,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 07:56:54,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 07:56:55,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 07:56:55,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 07:56:56,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 07:56:56,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 07:56:57,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 07:56:57,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 07:56:58,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 07:56:58,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 07:56:58,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 07:56:59,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 07:56:59,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 07:57:00,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 07:57:00,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 07:57:01,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 07:57:01,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 07:57:01,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 07:57:02,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 07:57:02,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 07:57:02,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 07:57:03,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 07:57:03,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 07:57:04,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 07:57:04,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 07:57:05,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 07:57:05,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 07:57:05,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 07:57:06,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 07:57:06,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 07:57:07,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 07:57:07,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 07:57:07,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 07:57:08,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 07:57:08,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 07:57:09,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 07:57:09,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 07:57:09,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 07:57:10,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 07:57:10,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 07:57:11,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 07:57:11,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 07:57:12,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 07:57:12,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 07:57:13,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 07:57:13,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 07:57:14,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 07:57:14,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 07:57:14,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 07:57:15,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 07:57:15,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 07:57:16,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 07:57:16,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 07:57:16,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 07:57:17,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 07:57:17,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 07:57:18,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 07:57:18,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 07:57:18,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 07:57:19,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 07:57:19,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 07:57:20,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 07:57:20,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 07:57:21,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 07:57:21,642 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 07:57:21,645 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 07:57:21,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:57:22,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 07:57:22,173 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:57:22,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 07:57:22,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 07:57:22,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 07:57:22,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 07:57:22,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 07:57:22,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 07:57:22,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 07:57:22,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 07:57:22,375 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-15 07:57:22,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 07:57:22,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 07:57:23,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 07:57:23,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 07:57:24,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 07:57:24,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 07:57:24,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 07:57:25,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 07:57:25,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 07:57:26,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 07:57:26,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 07:57:27,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 07:57:27,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 07:57:27,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 07:57:28,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 07:57:28,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 07:57:29,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 07:57:29,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 07:57:29,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 07:57:30,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 07:57:30,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 07:57:31,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 07:57:31,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 07:57:32,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 07:57:32,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 07:57:33,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 07:57:33,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 07:57:34,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 07:57:34,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 07:57:34,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 07:57:35,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 07:57:35,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 07:57:36,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 07:57:36,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 07:57:36,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 07:57:37,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 07:57:37,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 07:57:38,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 07:57:38,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 07:57:39,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 07:57:39,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 07:57:40,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 07:57:40,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 07:57:41,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 07:57:41,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 07:57:42,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 07:57:42,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 07:57:42,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 07:57:43,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 07:57:43,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 07:57:44,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 07:57:44,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 07:57:45,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 07:57:45,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 07:57:46,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 07:57:46,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 07:57:46,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 07:57:47,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 07:57:47,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 07:57:48,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 07:57:48,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 07:57:48,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 07:57:49,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 07:57:49,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 07:57:50,282 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 07:57:50,286 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 07:57:50,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:57:50,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 07:57:50,743 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:57:50,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 07:57:50,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 07:57:50,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 07:57:50,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 07:57:50,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 07:57:50,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 07:57:50,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 07:57:50,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 07:57:50,957 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-15 07:57:50,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 07:57:51,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 07:57:51,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 07:57:52,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 07:57:52,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 07:57:53,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 07:57:53,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 07:57:54,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 07:57:54,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 07:57:55,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 07:57:55,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 07:57:56,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 07:57:56,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 07:57:57,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 07:57:57,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 07:57:58,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 07:57:58,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 07:57:59,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 07:57:59,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 07:58:00,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 07:58:00,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 07:58:01,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 07:58:01,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 07:58:02,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 07:58:02,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 07:58:03,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 07:58:03,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 07:58:04,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 07:58:04,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 07:58:04,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 07:58:05,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 07:58:05,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 07:58:06,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 07:58:06,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 07:58:07,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 07:58:07,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 07:58:08,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 07:58:08,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 07:58:09,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 07:58:09,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 07:58:10,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 07:58:10,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 07:58:11,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 07:58:11,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 07:58:11,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 07:58:12,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 07:58:12,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 07:58:13,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 07:58:13,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 07:58:14,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 07:58:14,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 07:58:15,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 07:58:15,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 07:58:16,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 07:58:16,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 07:58:17,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 07:58:17,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 07:58:17,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 07:58:18,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 07:58:18,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 07:58:19,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 07:58:19,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 07:58:20,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 07:58:20,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 07:58:21,167 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 07:58:21,167 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 23845.533203125, u'ReplayWALOffset': u'7671:1536000', u'NTablesCreated': 588958, u'RecentReadIOAmpSet': 0.028743445873260498, u'ReadIOAmp': 307.0642395019531, u'BlockCacheMisses': 32576480, u'NFlushes': 150984, u'RecentWriteAmp': 2.728557586669922, u'NumSyncFlushes': 1242, u'TotalMemUsed': 134432669, u'NBloomFilterHits': 791009934, u'Fragmentation': 0.273274302482605, u'NumNormalFlushes': 78608, u'RecentBlockCacheHitRatio': 78.39569091796875, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 272087, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 6104088, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 423980756, u'CheckpointOverhead': 0, u'NItemsGet': 0, u'NWriteIOs': 434912, u'NReadBytes': 15595657876L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 19392749, u'SizeRatio': 0.42032651467756793, u'NWriteBytes': 720965632, u'BloomFilterMemUsed': 0, u'NumTables': 51, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 675840}, {u'TargetSize': 46137344, u'LogicalSize': 26429550, u'SizeRatio': 0.572845068844882, u'NWriteBytes': 580546560, u'BloomFilterMemUsed': 0, u'NumTables': 22, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 372736}, {u'TargetSize': 574463570, u'LogicalSize': 335026825, u'SizeRatio': 0.5831994272500169, u'NWriteBytes': 1555566592, u'BloomFilterMemUsed': 0, u'NumTables': 175, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 4079616}, {u'TargetSize': 29379438592L, u'LogicalSize': 16358540114L, u'SizeRatio': 0.5568023385734273, u'NWriteBytes': 1920548864, u'BloomFilterMemUsed': 0, u'NumTables': 492, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 154955776}, {u'TargetSize': 114892715210L, u'LogicalSize': 58758877184L, u'SizeRatio': 0.5114238711880121, u'NWriteBytes': 70855909376L, u'BloomFilterMemUsed': 0, u'NumTables': 243, u'WastedSpace': 1312519579, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 58758877184L}], u'NItemsGarbageCollected': 10695761, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 354160197, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 99552, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 5232088, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 292434, u'CompactWriter': {u'NDataBlockWrites': 1422062, u'NWriteBytes': 4319207424L, u'NIndexBlockWrites': 96406, u'NBytesIncoming': 5783360386L, u'NSyncs': 0, u'NInserts': 238061342, u'NWriteIOs': 92927}, u'NWriteBytes': 75684226509L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 10695761, u'LogicalDiskSize': 58758877184L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8115921767543816, u'BufferMemUsed': 498948, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 42400337070L, u'NTableFiles': 983, u'NFileCountCompacts': 0, u'TableObjectMemUsed': 257176, u'TreeSnapshotMemoryUsed': 141872, u'TotalDiskUsage': 70672211968L, u'BytesIncoming': 93254011900L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 30082, u'NItemsPut': 85736900, u'NSyncs': 72376, u'CompactReader': {u'NReadIOs': 5232088, u'NReadBytes': 15595657876L, u'NDataBlockReads': 5073929, u'NBytesOutgoing': 0, u'NIndexBlockReads': 158159}, u'ActiveDiskUsage': 58918961152L, u'NTablesDeleted': 128878, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 21619797, u'NWriteBytes': 71365019085L, u'NIndexBlockWrites': 571175, u'NBytesIncoming': 94019776505L, u'NSyncs': 72376, u'NInserts': 85732867, u'NWriteIOs': 341985}}, u'NReadBytes': 16124265151771L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 28.100000381469727, u'NInternalFragmentationCompacts': 33, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 378147008, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5037, u'NTxns': 6452155, u'NWriteBytes': 112556068864L, u'NSegments': 7672, u'NSyncs': 6452155, u'NLoggedBytes': 112556068864L, u'NActiveSegments': 2, u'NWrites': 6477016, u'NInserts': 122825708, u'DiskUsed': 16273408}, u'NGets': 19497057, u'NReadBytesSet': 37099462702L, u'NReadBytesCompact': 61188500633L, u'IndexResidentRatio': 0.045077938586473465, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.100000023841858, u'BlockCacheHitRatio': 71.3369369506836, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 16063076651138L, u'BloomFilterMemUsedRatio': 0.25824272632598877, u'NWriteBytes': 239079633122L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.12098872661590576, u'WriteAmp': 2.488827705383301, u'NBlocksCached': 26850, u'LogicalDataSize': 43854208184L, u'ActiveBloomFilterMemUsed': 38468768, u'NFileCountCompacts': 16900, u'NTableFiles': 1904, u'FlushQueueSize': 72, u'TableObjectMemUsed': 485584, u'BlockCacheQuota': 87589864, u'TotalDiskUsage': 72434601984L, u'NPurgerRuns': 132, u'ReadIOAmpSet': 0.0, u'NSyncs': 217128, u'NReadIOsSetPerSec': 4.933333396911621, u'NReadIOsPerSec': 28.766666412353516, u'NTablesDeleted': 445079, u'BloomFilterFPR': 0.016412697732448578, u'NBloomFilterHitsPerSec': 1774.5333251953125, u'WriteCacheMemUsed': 8744008, u'RecentBloomFilterFPR': 0.024087993428111076, u'BytesPerRead': 821969.0625, u'BytesIncomingPerSec': 325674.96875, u'NReadIOsGetPerSec': 9.300000190734863, u'NGetStats': 271846, u'TotalIndexBlocksSize': 448089989, u'NGetStatsPerSec': 3.366666555404663, u'NWriteBytesCompact': 48641589248L, u'NReadIOsSet': 13963228, u'ActiveDataSize': 43854208184L, u'NWriteBytesCompactPerSec': 46148.265625, u'NBloomFilterMissesPerSec': 43.79999923706055, u'NCommitBatches': 6452155, u'NGetStatsComputed': 271846, u'NWriteIOs': 7450335, u'NReadIOsGet': 6000812071L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21897466, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 171.63333129882812, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.7333333492279053, u'ManagedFileObjectMemUsed': 187968, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 84644032, u'NWriterCompacts': 0, u'NReadIOs': 6023272796L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1060257474, u'NTablesCreated': 165543, u'ReadIOAmp': 71.6017645871869, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 791009934, u'TotalIndexBlocksSize': 24098001, u'CheckpointOverhead': 7675904, u'NItemsGet': 83808157, u'NWriteIOs': 314703, u'NReadBytes': 16108519111295L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1857413, u'SizeRatio': 0.04025834256952459, u'NWriteBytes': 5194182656L, u'BloomFilterMemUsed': 24672, u'NumTables': 80, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1748992}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 25516104, u'SizeRatio': 0.5530466599897905, u'NWriteBytes': 19935727616L, u'BloomFilterMemUsed': 788896, u'NumTables': 70, u'WastedSpace': 34744, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 25473024}, {u'TargetSize': 158592276, u'LogicalSize': 132177234, u'SizeRatio': 0.8334405516697421, u'NWriteBytes': 10177273856L, u'BloomFilterMemUsed': 4025600, u'NumTables': 134, u'WastedSpace': 316337, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 132407296}, {u'TargetSize': 1585922801, u'LogicalSize': 1585922801, u'SizeRatio': 1.0, u'NWriteBytes': 14543400960L, u'BloomFilterMemUsed': 33628160, u'NumTables': 283, u'WastedSpace': 5452692, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1591373824}], u'NItemsGarbageCollected': 21425800, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 23981735, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 66432, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 6017738115L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 210313, u'CompactWriter': {u'NDataBlockWrites': 16148723, u'NWriteBytes': 44253552640L, u'NIndexBlockWrites': 210852, u'NBytesIncoming': 61494857492L, u'NSyncs': 0, u'NInserts': 1060257474, u'NWriteIOs': 99310}, u'NWriteBytes': 49562511262L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 6000811928L, u'NReadBytes': 16063076578027L, u'NDataBlockReads': 13242383448L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 25741170398L}, u'NBloomFilterMisses': 13199242, u'NItemsMerged': 0, u'LogicalDiskSize': 1585922801, u'TotalBloomFilterMemUsed': 38510504, u'WriteAmp': 15.727933503914798, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 38467328, u'LogicalDataSize': 1453745567, u'NTableFiles': 692, u'NFileCountCompacts': 8498, u'TableObjectMemUsed': 171616, u'TreeSnapshotMemoryUsed': 108560, u'TotalDiskUsage': 1760514048, u'BytesIncoming': 3151241150L, u'NItemsDel': 0, u'PendingCheckpointBytes': 3219456, u'NCompacts': 10838, u'NItemsPut': 64311100, u'NSyncs': 72376, u'CompactReader': {u'NReadIOs': 16926187, u'NReadBytes': 45442533268L, u'NDataBlockReads': 16580036, u'NBytesOutgoing': 0, u'NIndexBlockReads': 346151}, u'ActiveDiskUsage': 1751003136, u'NTablesDeleted': 164976, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.01641269758404249, u'Writer': {u'NDataBlockWrites': 1081575, u'NWriteBytes': 5308958622L, u'NIndexBlockWrites': 143212, u'NBytesIncoming': 3729807136L, u'NSyncs': 72376, u'NInserts': 64307067, u'NWriteIOs': 215393}}, u'NReadBytesCompactPerSec': 51320.30078125, u'TableMetaMemUsed': 568481, u'NDeletes': 0, u'BlockCacheHitsPerSec': 101.96666717529297, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 13199242, u'LogicalDiskSize': 60344925532L, u'NSets': 64311100, u'TotalBloomFilterMemUsed': 38511944, u'BufferMemUsed': 498948, u'RecentReadAmp': 0.0, u'NGetsPerSec': 76.86666870117188, u'NInserts': 0, u'NCommitBatchesPerSec': 149.3333282470703, u'FragmentationWithOverhead': 0.3945682644844055, u'NCompactsPerSec': 0.06666667014360428, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 75165.8359375, u'TreeSnapshotMemoryUsed': 294512, u'BytesIncoming': 96061147014L, u'NCompacts': 49322, u'NumPersistentFlushes': 71134, u'NWriteIOsPerSec': 152.86666870117188, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 310.21942138671875, u'ActiveDiskUsage': 60670808064L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 3.366666555404663, u'NReadBytesSetPerSec': 12270.2666015625, u'BlockCacheHits': 81076695, u'NWriteBytesPerSec': 888622.9375, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 151328, u'ReadIOAmp': 0.0064339062359398905, u'MemtablesMemUsed': 2270232, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 11232, u'CheckpointOverhead': 1032192, u'NItemsGet': 22226, u'NWriteIOs': 223704, u'NReadBytes': 150382600, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 833481, u'SizeRatio': 0.018065214157104492, u'NWriteBytes': 1170894848, u'BloomFilterMemUsed': 736, u'NumTables': 92, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 753664}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125547, u'SizeRatio': 0.0027211579409512606, u'NWriteBytes': 68829184, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 39150334, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5076, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 21984, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 302593, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 65734, u'CompactWriter': {u'NDataBlockWrites': 8402, u'NWriteBytes': 68829184, u'NIndexBlockWrites': 8402, u'NBytesIncoming': 38133428, u'NSyncs': 0, u'NInserts': 445306, u'NWriteIOs': 8402}, u'NWriteBytes': 1276826487, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 143, u'NReadBytes': 73111, u'NDataBlockReads': 120, u'NBytesOutgoing': 0, u'NIndexBlockReads': 120}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125547, u'TotalBloomFilterMemUsed': 1440, u'WriteAmp': 0.3310067033859863, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1440, u'LogicalDataSize': 125547, u'NTableFiles': 229, u'NFileCountCompacts': 8402, u'TableObjectMemUsed': 56792, u'TreeSnapshotMemoryUsed': 44080, u'TotalDiskUsage': 1875968, u'BytesIncoming': 3857403714L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 8402, u'NItemsPut': 39158157, u'NSyncs': 72376, u'CompactReader': {u'NReadIOs': 302450, u'NReadBytes': 150309489, u'NDataBlockReads': 151225, u'NBytesOutgoing': 0, u'NIndexBlockReads': 151225}, u'ActiveDiskUsage': 843776, u'NTablesDeleted': 151225, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 142926, u'NWriteBytes': 1207997303, u'NIndexBlockWrites': 142926, u'NBytesIncoming': 106781461, u'NSyncs': 72376, u'NInserts': 1397095, u'NWriteIOs': 215302}}}}} 2021-12-15 07:58:21,171 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.273274302482605, 0.2716084122657776, 0.2661140263080597, 0.27637410163879395, 0.27153128385543823, 0.27414020895957947, 0.264712393283844, 0.2696765661239624, 0.27368980646133423, 0.27168673276901245, 0.27053532004356384, 0.27214404940605164, 0.2660008370876312, 0.2696131467819214, 0.27153825759887695, 0.2716931700706482, 0.26946136355400085, 0.2709631323814392, 0.2741228938102722, 0.26823046803474426, 0.2722410559654236, 0.2736606001853943, 0.2712111473083496, 0.26895856857299805, 0.27144771814346313, 0.26993799209594727, 0.2732224762439728, 0.2672247588634491, 0.26907187700271606, 0.2685222923755646, 0.27044549584388733, 0.2705344259738922, 0.27180272340774536, 0.2674561142921448, 0.2726796269416809, 0.26979735493659973, 0.2730981111526489, 0.27215421199798584, 0.2709001898765564, 0.26798978447914124, 0.26954367756843567, 0.26532652974128723, 0.26549163460731506, 0.2733769118785858, 0.26622870564460754, 0.27223071455955505, 0.2663605511188507, 0.2685670554637909, 0.2664254605770111, 0.2709876298904419, 0.2705456614494324, 0.27328699827194214, 0.270804226398468, 0.27033042907714844, 0.2725986838340759, 0.2740248739719391, 0.27111437916755676, 0.27107229828834534, 0.2688163220882416, 0.27121952176094055, 0.2691781222820282, 0.27398255467414856, 0.2668136954307556, 0.26625439524650574], '172.23.110.66': [0.2714206576347351, 0.2721738815307617, 0.2726729214191437, 0.27244889736175537, 0.268398255109787, 0.27711352705955505, 0.2725059688091278, 0.26801156997680664, 0.2689947783946991, 0.27100154757499695, 0.27014195919036865, 0.2750336229801178, 0.27087047696113586, 0.2695668041706085, 0.2733610272407532, 0.2710496485233307, 0.26742836833000183, 0.274562269449234, 0.2745487689971924, 0.2731092870235443, 0.2708023488521576, 0.2715665102005005, 0.27193543314933777, 0.27290794253349304, 0.267731249332428, 0.27413463592529297, 0.27333927154541016, 0.27096986770629883, 0.27486640214920044, 0.27054253220558167, 0.2755182683467865, 0.27041420340538025, 0.27478212118148804, 0.27330097556114197, 0.26911506056785583, 0.27257585525512695, 0.27140483260154724, 0.27367788553237915, 0.27226200699806213, 0.2694939970970154, 0.2690386176109314, 0.26824676990509033, 0.2729576826095581, 0.2741888165473938, 0.2713581919670105, 0.2726179361343384, 0.27639973163604736, 0.276457816362381, 0.2688748240470886, 0.2736717760562897, 0.2677103877067566, 0.27361029386520386, 0.27212807536125183, 0.2739507853984833, 0.2749576270580292, 0.265007346868515, 0.270438551902771, 0.27221301198005676, 0.26951560378074646, 0.27298682928085327, 0.26661744713783264, 0.26954972743988037, 0.2653992176055908, 0.2701195180416107], '172.23.110.64': [0.27213263511657715, 0.26668670773506165, 0.2683941721916199, 0.26544371247291565, 0.27098479866981506, 0.27052322030067444, 0.2739737629890442, 0.2735230028629303, 0.2657954692840576, 0.2690460681915283, 0.2640332579612732, 0.27149325609207153, 0.26300421357154846, 0.26848405599594116, 0.2711801528930664, 0.2708316445350647, 0.26536473631858826, 0.2671976387500763, 0.26713186502456665, 0.26397767663002014, 0.2715989351272583, 0.26648882031440735, 0.26805028319358826, 0.27224379777908325, 0.2670750319957733, 0.27069175243377686, 0.2751390039920807, 0.27581876516342163, 0.26824554800987244, 0.26811155676841736, 0.27320703864097595, 0.2729828655719757, 0.26577889919281006, 0.2648261487483978, 0.2760409116744995, 0.26482561230659485, 0.2621528208255768, 0.27059322595596313, 0.26666513085365295, 0.27235496044158936, 0.2699483335018158, 0.2730409502983093, 0.2717914283275604, 0.2685718536376953, 0.2664140462875366, 0.27069470286369324, 0.2703189551830292, 0.27206820249557495, 0.2707839608192444, 0.2691776752471924, 0.27165842056274414, 0.26336154341697693, 0.2665257155895233, 0.2659846842288971, 0.271634042263031, 0.2648460865020752, 0.2716487646102905, 0.26915276050567627, 0.2731272280216217, 0.27004173398017883, 0.26683834195137024, 0.26874271035194397, 0.27270933985710144, 0.26762834191322327]} 2021-12-15 07:58:21,533 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 27.0 2021-12-15 07:58:21,891 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 27.0 2021-12-15 07:58:22,247 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 27.0 2021-12-15 07:58:22,247 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 27.0, '172.23.110.66': 27.0, '172.23.110.64': 27.0} 2021-12-15 07:58:22,250 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 07:58:22,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:58:22,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 07:58:22,707 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:58:22,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 07:58:22,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 07:58:22,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 07:58:22,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 07:58:22,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 07:58:22,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 07:58:22,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 07:58:22,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 07:58:22,871 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 07:58:22,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 07:58:22,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:22,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 07:58:23,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:23,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:23,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:23,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 07:58:23,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 07:58:23,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 07:58:23,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 07:58:23,247 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 07:58:23,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 07:58:23,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:23,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 07:58:23,302 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 07:58:23,303 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 07:58:23,306 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 07:58:23,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:58:23,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 07:58:23,822 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:58:23,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 07:58:23,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 07:58:23,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 07:58:23,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 07:58:23,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 07:58:23,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 07:58:23,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 07:58:23,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 07:58:24,006 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 07:58:24,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 07:58:24,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:24,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 07:58:24,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:24,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:24,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:24,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 07:58:24,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 07:58:24,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 07:58:24,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 07:58:24,345 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 07:58:24,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 07:58:24,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:24,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 07:58:24,395 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 07:58:24,395 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 07:58:24,398 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 07:58:24,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 07:58:24,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 07:58:24,923 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 07:58:24,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 07:58:24,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 07:58:24,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 07:58:24,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 07:58:25,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 07:58:25,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 07:58:25,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 07:58:25,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 07:58:25,065 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 07:58:25,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 07:58:25,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:25,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 07:58:25,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:25,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 07:58:25,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 07:58:25,299 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 07:58:25,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 07:58:25,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 07:58:25,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 07:58:25,355 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 07:58:25,355 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # Step 3.1: Update 40000000 RandonKey keys to create 50 percent fragmentation # ############################################################ ############################################################ # # RAM FootPrint: 6354.421875 # ############################################################ Read Start: 0 Read End: 0 Update Start: 40000000 Update End: 80000000 Expiry Start: 0 Expiry End: 0 Delete Start: 0 Delete End: 0 Create Start: 0 Create End: 0 Final Start: 40000000 Final End: 80000000 2021-12-15 07:58:25,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:25,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1639583906.4 2021-12-15 07:58:26,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:26,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1639583907.41 2021-12-15 07:58:27,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:27,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1639583908.42 2021-12-15 07:58:28,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:28,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1639583909.44 2021-12-15 07:58:29,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:29,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1639583910.45 2021-12-15 07:58:30,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:30,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1639583911.46 2021-12-15 07:58:31,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:31,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1639583912.47 2021-12-15 07:58:32,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:32,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1639583913.48 2021-12-15 07:58:33,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:33,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1639583914.5 2021-12-15 07:58:34,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:34,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1639583915.57 2021-12-15 07:58:35,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:35,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:36,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:36,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:37,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:37,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:38,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:38,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:39,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:39,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:40,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:40,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:41,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:41,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:42,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:42,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:43,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:43,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:44,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:44,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:45,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:45,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:46,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:46,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:47,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:47,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:48,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:48,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:49,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:49,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:50,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:50,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:51,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:51,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:52,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:52,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:53,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:53,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:54,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:54,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:55,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:55,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:56,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:56,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:57,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:57,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:58,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:58,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:58:59,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:58:59,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:00,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:00,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:01,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:01,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:02,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:02,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:03,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:03,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:04,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:04,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:05,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:05,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:07,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:07,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:08,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:08,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:09,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:09,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:10,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:10,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:11,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:11,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:12,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:12,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:13,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:13,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:14,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:14,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 07:59:15,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 07:59:15,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Loader_GleamBookUsers0__default_VolumeCollection45_7_1639583909.44 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection35_0_1639583916.58 Loader_GleamBookUsers0__default_VolumeCollection42_4_1639583912.47 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639583917.59 Loader_GleamBookUsers0__default_VolumeCollection44_8_1639583908.42 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639583918.6 Loader_GleamBookUsers0__default_VolumeCollection47_9_1639583907.41 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639583919.62 Loader_GleamBookUsers0__default_VolumeCollection49_5_1639583911.46 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639583920.63 Loader_GleamBookUsers0__default_VolumeCollection48_6_1639583910.45 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639583921.65 Loader_GleamBookUsers0__default_VolumeCollection40_2_1639583914.5 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639583922.66 Loader_GleamBookUsers0__default_VolumeCollection46_10_1639583906.4 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639583923.67 Loader_GleamBookUsers0__default_VolumeCollection43_3_1639583913.48 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639583924.68 Loader_GleamBookUsers0__default_VolumeCollection41_1_1639583915.57 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639583925.69 Loader_GleamBookUsers0__default_VolumeCollection35_0_1639583916.58 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639583926.71 Loader_GleamBookUsers0__default_VolumeCollection33_-2_1639583918.6 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639583927.73 Loader_GleamBookUsers0__default_VolumeCollection36_-1_1639583917.59 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639583928.74 Loader_GleamBookUsers0__default_VolumeCollection39_-4_1639583920.63 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639583929.76 Loader_GleamBookUsers0__default_VolumeCollection34_-3_1639583919.62 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639583930.77 Loader_GleamBookUsers0__default_VolumeCollection38_-6_1639583922.66 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639583931.78 Loader_GleamBookUsers0__default_VolumeCollection6_-8_1639583924.68 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639583932.79 Loader_GleamBookUsers0__default_VolumeCollection7_-7_1639583923.67 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639583933.8 Loader_GleamBookUsers0__default_VolumeCollection37_-5_1639583921.65 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639583934.81 Loader_GleamBookUsers0__default_VolumeCollection5_-9_1639583925.69 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639583935.83 Loader_GleamBookUsers0__default_VolumeCollection4_-10_1639583926.71 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639583936.84 Loader_GleamBookUsers0__default_VolumeCollection3_-11_1639583927.73 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639583937.85 Loader_GleamBookUsers0__default_VolumeCollection2_-13_1639583929.76 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639583938.86 Loader_GleamBookUsers0__default_VolumeCollection31_-12_1639583928.74 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639583939.9 Loader_GleamBookUsers0__default_VolumeCollection1_-15_1639583931.78 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639583940.92 Loader_GleamBookUsers0__default_VolumeCollection32_-14_1639583930.77 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639583941.93 Loader_GleamBookUsers0__default_VolumeCollection0_-16_1639583932.79 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639583942.94 Loader_GleamBookUsers0__default_VolumeCollection8_-19_1639583935.83 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639583943.96 Loader_GleamBookUsers0__default_VolumeCollection9_-18_1639583934.81 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639583944.97 Loader_GleamBookUsers0__default_VolumeCollection30_-17_1639583933.8 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639583945.98 Loader_GleamBookUsers0__default_VolumeCollection24_-21_1639583937.85 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639583946.99 Loader_GleamBookUsers0__default_VolumeCollection22_-23_1639583939.9 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639583948.01 Loader_GleamBookUsers0__default_VolumeCollection19_-20_1639583936.84 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639583949.02 Loader_GleamBookUsers0__default_VolumeCollection25_-22_1639583938.86 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639583950.03 Loader_GleamBookUsers0__default_VolumeCollection28_-25_1639583941.93 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639583951.05 Loader_GleamBookUsers0__default_VolumeCollection26_-27_1639583943.96 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639583952.06 Loader_GleamBookUsers0__default_VolumeCollection23_-24_1639583940.92 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639583953.09 Loader_GleamBookUsers0__default_VolumeCollection29_-26_1639583942.94 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639583954.1 Loader_GleamBookUsers0__default_VolumeCollection20_-29_1639583945.98 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639583955.17 Loader_GleamBookUsers0__default_VolumeCollection27_-28_1639583944.97 is completed! Starting Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639583956.19 Loader_GleamBookUsers0__default_VolumeCollection13_-31_1639583948.01 is completed! Loader_GleamBookUsers0__default_VolumeCollection21_-30_1639583946.99 is completed! Loader_GleamBookUsers0__default_VolumeCollection14_-32_1639583949.02 is completed! Loader_GleamBookUsers0__default_VolumeCollection11_-33_1639583950.03 is completed! Loader_GleamBookUsers0__default_VolumeCollection12_-34_1639583951.05 is completed! Loader_GleamBookUsers0__default_VolumeCollection17_-35_1639583952.06 is completed! Loader_GleamBookUsers0__default_VolumeCollection18_-36_1639583953.09 is completed! Loader_GleamBookUsers0__default_VolumeCollection16_-38_1639583955.17 is completed! Loader_GleamBookUsers0__default_VolumeCollection15_-37_1639583954.1 is completed! Loader_GleamBookUsers0__default_VolumeCollection10_-39_1639583956.19 is completed! 2021-12-15 11:51:13,381 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:649] All nodes in cluster: [ip:172.23.110.64 port:8091 ssh_username:root, ip:172.23.110.66 port:8091 ssh_username:root, ip:172.23.110.65 port:8091 ssh_username:root] 2021-12-15 11:51:13,388 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 11:51:13,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:13,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 11:51:13,849 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:13,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 11:51:13,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 11:51:13,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 11:51:13,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 11:51:13,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 11:51:13,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 11:51:13,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 11:51:14,002 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 11:51:14,003 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.0 2021-12-15 11:51:14,003 | infra | INFO | pool-3-thread-17 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.0' started 2021-12-15 11:51:14,006 | infra | DEBUG | pool-3-thread-17 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 11:51:14,007 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 11:51:14,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:14,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 11:51:14,513 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:14,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 11:51:14,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 11:51:14,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 11:51:14,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 11:51:14,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 11:51:14,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 11:51:14,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 11:51:14,635 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 11:51:14,637 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.64 2021-12-15 11:51:14,638 | infra | INFO | pool-3-thread-18 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.64' started 2021-12-15 11:51:14,638 | infra | DEBUG | pool-3-thread-18 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 11:51:14,640 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 11:51:14,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:14,812 | test | DEBUG | pool-3-thread-17 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 11:51:14,812 | test | DEBUG | pool-3-thread-17 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.64': 0} for bucket 'GleamBookUsers0' 2021-12-15 11:51:14,813 | infra | INFO | pool-3-thread-17 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.0' completed 2021-12-15 11:51:15,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 11:51:15,168 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:15,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 11:51:15,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 11:51:15,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 11:51:15,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 11:51:15,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 11:51:15,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 11:51:15,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 11:51:15,303 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket GleamBookUsers0 2021-12-15 11:51:15,305 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597875.3 2021-12-15 11:51:15,305 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.0 2021-12-15 11:51:15,306 | infra | INFO | pool-3-thread-19 | [task:start_task:90] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597875.3' started 2021-12-15 11:51:15,306 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 11:51:15,308 | infra | DEBUG | pool-3-thread-19 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 checkpoint 2021-12-15 11:51:15,309 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.64 2021-12-15 11:51:15,461 | test | DEBUG | pool-3-thread-18 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 11:51:15,461 | test | DEBUG | pool-3-thread-18 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.65': 0} for bucket 'GleamBookUsers0' 2021-12-15 11:51:15,461 | infra | INFO | pool-3-thread-18 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597874.64' completed 2021-12-15 11:51:15,463 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 11:51:15,463 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597875.3 2021-12-15 11:51:16,086 | test | DEBUG | pool-3-thread-19 | [task:_compare:2726] Comparing 0 == 0 2021-12-15 11:51:16,086 | test | DEBUG | pool-3-thread-19 | [task:_get_checkpoint_stats_and_compare:2715] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.110.66': 0} for bucket 'GleamBookUsers0' 2021-12-15 11:51:16,086 | infra | INFO | pool-3-thread-19 | [task:complete_task:105] Thread 'StatsWaitTask_GleamBookUsers0_num_items_for_persistence_1639597875.3' completed 2021-12-15 11:51:16,088 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 11:51:16,130 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 19710.7, u'diskFetches': 0, u'dataUsed': 7366072232650L, u'itemCount': 4000000000L, u'quotaPercentUsed': 77.44291344036658, u'storageTotals': {u'hdd': {u'usedByData': 14073593650482L, u'used': 14197589708631L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 21778220282025L}, u'ram': {u'usedByData': 249461085400L, u'used': 415405998080L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 14073593650482L, u'vbActiveNumNonResident': 3867098126L, u'memUsed': 249461085400L} 2021-12-15 11:51:16,505 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4000000000 | 300.00 GiB | 232.33 GiB | 12.80 TiB | 3.32254685 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+------------+ 2021-12-15 11:51:16,506 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+-------------------+---------------+---------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+-------------------+---------------+---------------+---------------+ | 4000000000 | 4000000000 | 40000000-80000000 | 0-0 | 0-0 | 0-0 | +---------------+---------------+-------------------+---------------+---------------+---------------+ 2021-12-15 11:51:16,872 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.32254685 2021-12-15 11:51:16,875 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 11:51:16,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:17,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 11:51:17,332 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:17,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 11:51:17,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 11:51:17,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 11:51:17,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 11:51:17,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 11:51:17,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 11:51:17,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 11:51:17,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 11:51:18,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 11:51:18,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 11:51:18,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 11:51:18,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 11:51:21,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 337 2021-12-15 11:51:21,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 390 2021-12-15 11:51:21,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-360 is 356 2021-12-15 11:51:21,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 330 2021-12-15 11:51:21,388 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-730 is 322 2021-12-15 11:51:21,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 365 2021-12-15 11:51:21,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 308 2021-12-15 11:51:21,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 332 2021-12-15 11:51:21,390 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 309 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 337 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 302 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 346 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 317 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-86 is 303 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 374 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-735 is 330 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 309 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 351 2021-12-15 11:51:21,391 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 303 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 379 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-306 is 300 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 320 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-163 is 306 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-794 is 305 2021-12-15 11:51:21,392 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 302 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-146 is 414 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 303 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-309 is 307 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-165 is 344 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-815 is 344 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-743 is 315 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-168 is 339 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-96 is 345 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-313 is 304 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-98 is 309 2021-12-15 11:51:21,394 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-388 is 301 2021-12-15 11:51:21,395 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-172 is 358 2021-12-15 11:51:21,395 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 367 2021-12-15 11:51:21,395 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 309 2021-12-15 11:51:21,395 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 308 2021-12-15 11:51:21,395 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-363 is 300 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-822 is 330 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 324 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-752 is 343 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 396 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-105 is 302 2021-12-15 11:51:21,397 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 300 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 307 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-106 is 302 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 303 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-396 is 312 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-757 is 300 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-325 is 322 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 302 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 302 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-724 is 305 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-148 is 361 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 349 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-328 is 321 2021-12-15 11:51:21,398 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 328 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 366 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 303 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-332 is 329 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 302 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-693 is 309 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-405 is 351 2021-12-15 11:51:21,400 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-333 is 304 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-117 is 396 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 307 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-766 is 437 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 326 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-479 is 353 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 320 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-264 is 302 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 310 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-121 is 313 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 304 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-365 is 415 2021-12-15 11:51:21,401 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 329 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-338 is 333 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-124 is 343 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 317 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 316 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 377 2021-12-15 11:51:21,403 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-703 is 382 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 374 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-271 is 303 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 303 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 373 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 337 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 301 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 302 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-344 is 307 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-705 is 332 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 329 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-706 is 348 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-130 is 311 2021-12-15 11:51:21,404 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 301 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-131 is 315 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 317 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-366 is 316 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 304 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-205 is 349 2021-12-15 11:51:21,405 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 300 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 302 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-712 is 332 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-281 is 330 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-426 is 353 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 312 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 320 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 315 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-284 is 348 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-140 is 362 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 343 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 300 2021-12-15 11:51:21,407 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-143 is 302 2021-12-15 11:51:21,408 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 306 2021-12-15 11:51:21,408 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-369 is 338 2021-12-15 11:51:21,408 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 318 2021-12-15 11:51:21,408 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 328 2021-12-15 11:51:21,408 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 11:51:21,413 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 11:51:21,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:21,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 11:51:21,887 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:21,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 11:51:21,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 11:51:21,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 11:51:21,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 11:51:21,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 11:51:21,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 11:51:21,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 11:51:22,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 11:51:22,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 11:51:22,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 11:51:22,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 11:51:23,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 11:51:25,665 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-648 is 374 2021-12-15 11:51:25,665 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-360 is 334 2021-12-15 11:51:25,665 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 305 2021-12-15 11:51:25,665 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-361 is 307 2021-12-15 11:51:25,665 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-73 is 302 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 300 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-874 is 372 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 303 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-876 is 386 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-589 is 323 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-1022 is 365 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-950 is 382 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-374 is 342 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-375 is 314 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-159 is 356 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-88 is 307 2021-12-15 11:51:25,667 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-449 is 305 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-666 is 304 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-162 is 312 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 304 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-668 is 317 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-885 is 336 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-526 is 301 2021-12-15 11:51:25,668 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-671 is 344 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-456 is 365 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-889 is 313 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-385 is 301 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-386 is 300 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-27 is 301 2021-12-15 11:51:25,670 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-532 is 307 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-605 is 329 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-579 is 347 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-507 is 345 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-147 is 358 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-606 is 357 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-390 is 301 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-102 is 356 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-967 is 361 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-679 is 373 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-535 is 300 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-897 is 364 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-609 is 328 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-465 is 328 2021-12-15 11:51:25,671 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-393 is 320 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-970 is 331 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-682 is 301 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-394 is 315 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-106 is 394 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-971 is 349 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-899 is 332 2021-12-15 11:51:25,673 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-539 is 330 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-612 is 305 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-540 is 337 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-468 is 431 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-973 is 305 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-613 is 318 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-469 is 338 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-37 is 311 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-902 is 365 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-542 is 387 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-470 is 314 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-110 is 302 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-38 is 361 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-903 is 309 2021-12-15 11:51:25,674 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-543 is 384 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-868 is 303 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-436 is 307 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-4 is 366 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-400 is 321 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-112 is 370 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-40 is 341 2021-12-15 11:51:25,676 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-545 is 320 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-618 is 345 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-402 is 381 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-42 is 327 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-620 is 321 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 385 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-476 is 310 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 304 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-981 is 415 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-909 is 324 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-621 is 395 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-549 is 316 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 370 2021-12-15 11:51:25,677 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-622 is 308 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-912 is 349 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-624 is 386 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-552 is 312 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-625 is 307 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-481 is 313 2021-12-15 11:51:25,678 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-869 is 385 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-365 is 310 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-914 is 332 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-626 is 376 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-554 is 321 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-483 is 323 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-411 is 300 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-123 is 352 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 370 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-484 is 304 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-124 is 328 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-989 is 348 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-629 is 340 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-413 is 307 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-53 is 318 2021-12-15 11:51:25,680 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-630 is 305 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-558 is 317 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-342 is 343 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 302 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-991 is 303 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-55 is 340 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-560 is 352 2021-12-15 11:51:25,681 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-416 is 304 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-921 is 367 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-994 is 309 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-922 is 321 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-346 is 336 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-635 is 339 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-491 is 315 2021-12-15 11:51:25,683 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-131 is 327 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-637 is 308 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-349 is 349 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-133 is 300 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-854 is 324 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-567 is 331 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-640 is 333 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-424 is 305 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-931 is 305 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-643 is 309 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-571 is 392 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-499 is 300 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-139 is 311 2021-12-15 11:51:25,684 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-860 is 375 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-1005 is 349 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-141 is 347 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-1015 is 305 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-943 is 351 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-439 is 304 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-7 is 339 2021-12-15 11:51:25,686 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-1006 is 377 2021-12-15 11:51:25,687 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-502 is 310 2021-12-15 11:51:25,687 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-440 is 304 2021-12-15 11:51:25,687 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 11:51:25,690 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 11:51:25,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:26,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 11:51:26,214 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:26,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 11:51:26,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 11:51:26,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 11:51:26,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 11:51:26,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 11:51:26,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 11:51:26,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 11:51:26,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 11:51:26,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 11:51:26,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 11:51:27,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 11:51:27,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 11:51:30,150 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-864 is 327 2021-12-15 11:51:30,150 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-289 is 380 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-730 is 324 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-298 is 302 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-732 is 343 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-660 is 300 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-228 is 307 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-1021 is 315 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-517 is 307 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-806 is 325 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-518 is 320 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-303 is 326 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-664 is 318 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-304 is 342 2021-12-15 11:51:30,151 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-232 is 305 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-737 is 308 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-665 is 311 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-810 is 358 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-666 is 316 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-522 is 305 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-523 is 303 2021-12-15 11:51:30,153 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-938 is 335 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-866 is 330 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-596 is 300 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-308 is 398 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-236 is 349 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-670 is 321 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-526 is 307 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-743 is 359 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-744 is 307 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-961 is 389 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-889 is 334 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-241 is 302 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-962 is 309 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-530 is 300 2021-12-15 11:51:30,154 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-819 is 301 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-965 is 308 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-317 is 300 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-245 is 301 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-291 is 315 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-219 is 311 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-894 is 310 2021-12-15 11:51:30,155 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-678 is 348 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-606 is 306 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-751 is 338 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-896 is 302 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-609 is 307 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-898 is 310 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-754 is 304 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-178 is 305 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-755 is 310 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-539 is 314 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-323 is 306 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-251 is 316 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-972 is 300 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-756 is 329 2021-12-15 11:51:30,157 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-684 is 303 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-757 is 301 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-902 is 308 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-830 is 305 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-614 is 348 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-759 is 381 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-687 is 303 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-615 is 304 2021-12-15 11:51:30,158 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-543 is 305 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-327 is 349 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-652 is 305 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-292 is 305 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-976 is 312 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-904 is 304 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-688 is 310 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-617 is 311 2021-12-15 11:51:30,160 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-329 is 301 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-257 is 312 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-185 is 348 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.42/kvstore-618 is 330 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-764 is 313 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-620 is 331 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-981 is 316 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-909 is 305 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-765 is 300 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-911 is 311 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-695 is 310 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-623 is 307 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-335 is 308 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-263 is 338 2021-12-15 11:51:30,161 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-984 is 300 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-768 is 302 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-552 is 367 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-192 is 302 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-913 is 304 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-265 is 302 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-869 is 303 2021-12-15 11:51:30,163 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-770 is 311 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-554 is 303 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-771 is 350 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-627 is 302 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-195 is 304 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-988 is 300 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-772 is 303 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-989 is 310 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-990 is 310 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-702 is 351 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-992 is 318 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-704 is 326 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-560 is 359 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-272 is 315 2021-12-15 11:51:30,164 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-561 is 346 2021-12-15 11:51:30,165 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-726 is 307 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-654 is 306 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-294 is 302 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-708 is 309 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-997 is 341 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-637 is 303 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-566 is 306 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-206 is 305 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-567 is 350 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-640 is 323 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-280 is 302 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-208 is 323 2021-12-15 11:51:30,167 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-930 is 368 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-859 is 317 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-932 is 311 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-860 is 302 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-861 is 300 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-573 is 356 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-943 is 304 2021-12-15 11:51:30,168 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-655 is 311 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-862 is 351 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-718 is 306 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-646 is 302 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-286 is 310 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-1007 is 300 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-791 is 313 2021-12-15 11:51:30,170 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-575 is 302 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-872 is 310 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-800 is 309 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-224 is 317 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-1017 is 301 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-945 is 316 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-657 is 300 2021-12-15 11:51:30,171 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-513 is 348 2021-12-15 11:51:30,171 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 11:51:30,171 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 13455692MB 2021-12-15 11:51:30,171 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 7077MB 2021-12-15 11:51:30,171 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 379001MB 2021-12-15 11:51:30,171 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 13076281MB 2021-12-15 11:51:30,176 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 11:51:30,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:30,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 11:51:30,684 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:30,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 11:51:30,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 11:51:30,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 11:51:30,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 11:51:30,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 11:51:30,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 11:51:30,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 11:51:30,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 11:51:30,904 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-15 11:51:30,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 11:51:31,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 11:51:31,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 11:51:32,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 11:51:32,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 11:51:33,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 11:51:33,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 11:51:33,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 11:51:34,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 11:51:34,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 11:51:35,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 11:51:35,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 11:51:36,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 11:51:36,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 11:51:37,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 11:51:37,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 11:51:38,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 11:51:38,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 11:51:38,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 11:51:39,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 11:51:39,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 11:51:40,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 11:51:40,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 11:51:41,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 11:51:41,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 11:51:41,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 11:51:42,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 11:51:42,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 11:51:43,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 11:51:43,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 11:51:44,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 11:51:44,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 11:51:45,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 11:51:45,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 11:51:45,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 11:51:46,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 11:51:46,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 11:51:47,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 11:51:47,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 11:51:48,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 11:51:48,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 11:51:49,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 11:51:49,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 11:51:49,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 11:51:50,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 11:51:50,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 11:51:51,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 11:51:51,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 11:51:52,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 11:51:52,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 11:51:52,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 11:51:53,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 11:51:53,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 11:51:54,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 11:51:54,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 11:51:54,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 11:51:55,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 11:51:55,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 11:51:56,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 11:51:56,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 11:51:56,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 11:51:57,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 11:51:57,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 11:51:58,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 11:51:58,674 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 11:51:58,677 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 11:51:58,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:51:59,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 11:51:59,211 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:51:59,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 11:51:59,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 11:51:59,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 11:51:59,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 11:51:59,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 11:51:59,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 11:51:59,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 11:51:59,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 11:51:59,420 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-15 11:51:59,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 11:51:59,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 11:52:00,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 11:52:00,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 11:52:01,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 11:52:01,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 11:52:02,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 11:52:02,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 11:52:03,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 11:52:03,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 11:52:03,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 11:52:04,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 11:52:04,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 11:52:05,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 11:52:05,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 11:52:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 11:52:06,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 11:52:06,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 11:52:07,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 11:52:07,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 11:52:08,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 11:52:08,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 11:52:09,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 11:52:09,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 11:52:10,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 11:52:10,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 11:52:10,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 11:52:11,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 11:52:11,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 11:52:11,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 11:52:12,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 11:52:12,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 11:52:13,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 11:52:13,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 11:52:14,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 11:52:14,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 11:52:15,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 11:52:15,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 11:52:15,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 11:52:16,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 11:52:16,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 11:52:17,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 11:52:17,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 11:52:18,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 11:52:18,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 11:52:19,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 11:52:19,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 11:52:20,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 11:52:20,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 11:52:21,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 11:52:21,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 11:52:22,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 11:52:22,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 11:52:22,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 11:52:23,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 11:52:23,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 11:52:24,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 11:52:24,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 11:52:24,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 11:52:25,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 11:52:25,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 11:52:26,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 11:52:26,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 11:52:27,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 11:52:27,737 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 11:52:27,740 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 11:52:27,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:52:28,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 11:52:28,210 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:52:28,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 11:52:28,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 11:52:28,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 11:52:28,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 11:52:28,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 11:52:28,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 11:52:28,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 11:52:28,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 11:52:28,463 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-15 11:52:28,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 11:52:28,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 11:52:29,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 11:52:29,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 11:52:30,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 11:52:30,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 11:52:31,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 11:52:31,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 11:52:32,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 11:52:32,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 11:52:33,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 11:52:33,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 11:52:34,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 11:52:34,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 11:52:35,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 11:52:35,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 11:52:36,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 11:52:36,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 11:52:37,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 11:52:37,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 11:52:38,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 11:52:38,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 11:52:39,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 11:52:39,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 11:52:40,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 11:52:40,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 11:52:41,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 11:52:41,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 11:52:42,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 11:52:42,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 11:52:42,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 11:52:43,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 11:52:43,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 11:52:44,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 11:52:44,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 11:52:45,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 11:52:45,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 11:52:46,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 11:52:46,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 11:52:47,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 11:52:47,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 11:52:48,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 11:52:48,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 11:52:49,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 11:52:49,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 11:52:50,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 11:52:50,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 11:52:51,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 11:52:51,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 11:52:52,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 11:52:52,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 11:52:53,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 11:52:53,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 11:52:53,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 11:52:54,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 11:52:54,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 11:52:55,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 11:52:55,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 11:52:56,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 11:52:56,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 11:52:57,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 11:52:57,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 11:52:58,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 11:52:58,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 11:52:59,335 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 11:52:59,335 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 17883.466796875, u'ReplayWALOffset': u'9446:13283328', u'NTablesCreated': 775886, u'RecentReadIOAmpSet': 0.03931500390172005, u'ReadIOAmp': 1682.5240478515625, u'BlockCacheMisses': 57458386, u'NFlushes': 186225, u'RecentWriteAmp': 3.0262463092803955, u'NumSyncFlushes': 2037, u'TotalMemUsed': 130842223, u'NBloomFilterHits': 974116485, u'Fragmentation': 0.4711271822452545, u'NumNormalFlushes': 101343, u'RecentBlockCacheHitRatio': 92.16471099853516, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 416601, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 1680408, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 591611329, u'CheckpointOverhead': 7659520, u'NItemsGet': 0, u'NWriteIOs': 635830, u'NReadBytes': 31683635027L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 26279063, u'SizeRatio': 0.5695833509618585, u'NWriteBytes': 1325617152, u'BloomFilterMemUsed': 0, u'NumTables': 94, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1101824}, {u'TargetSize': 46137344, u'LogicalSize': 30227618, u'SizeRatio': 0.6551659757440741, u'NWriteBytes': 1134227456, u'BloomFilterMemUsed': 0, u'NumTables': 22, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 405504}, {u'TargetSize': 716409455, u'LogicalSize': 693037055, u'SizeRatio': 0.9673756399543889, u'NWriteBytes': 4232306688L, u'BloomFilterMemUsed': 0, u'NumTables': 236, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 7974912}, {u'TargetSize': 36475232256L, u'LogicalSize': 35027280062L, u'SizeRatio': 0.9603031398446594, u'NWriteBytes': 4749271040L, u'BloomFilterMemUsed': 0, u'NumTables': 1057, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 346800128}, {u'TargetSize': 143281892150L, u'LogicalSize': 72950464512L, u'SizeRatio': 0.5091394552190104, u'NWriteBytes': 95861116928L, u'BloomFilterMemUsed': 0, u'NumTables': 236, u'WastedSpace': 1309518437, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 72950464512L}], u'NItemsGarbageCollected': 18910168, u'NInternalFragmentationCompacts': 45, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 447657456, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 179520, u'NNonL0LocalCompacts': 6, u'NWriterCompacts': 0, u'NReadIOs': 10733795, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 527340, u'CompactWriter': {u'NDataBlockWrites': 3965475, u'NWriteBytes': 11945263104L, u'NIndexBlockWrites': 216397, u'NBytesIncoming': 16183506117L, u'NSyncs': 0, u'NInserts': 603890472, u'NWriteIOs': 207038}, u'NWriteBytes': 107225682655L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 18910168, u'LogicalDiskSize': 72950464512L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8608592523468093, u'BufferMemUsed': 494712, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 37923184450L, u'NTableFiles': 1816, u'NFileCountCompacts': 6, u'TableObjectMemUsed': 463760, u'TreeSnapshotMemoryUsed': 205984, u'TotalDiskUsage': 96421818368L, u'BytesIncoming': 124556578050L, u'NItemsDel': 0, u'PendingCheckpointBytes': 93954048, u'NCompacts': 59284, u'NItemsPut': 128654800, u'NSyncs': 84882, u'CompactReader': {u'NReadIOs': 10733795, u'NReadBytes': 31683635027L, u'NDataBlockReads': 10418855, u'NBytesOutgoing': 0, u'NIndexBlockReads': 314940}, u'ActiveDiskUsage': 73306746880L, u'NTablesDeleted': 254517, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 28906041, u'NWriteBytes': 95280419551L, u'NIndexBlockWrites': 693757, u'NBytesIncoming': 125713035522L, u'NSyncs': 84882, u'NInserts': 128652856, u'NWriteIOs': 428792}}, u'NReadBytes': 87740839465053L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 38.5, u'NInternalFragmentationCompacts': 45, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 474332180, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 5037, u'NTxns': 6867145, u'NWriteBytes': 145460699136L, u'NSegments': 9449, u'NSyncs': 6867145, u'NLoggedBytes': 145460699136L, u'NActiveSegments': 4, u'NWrites': 6899928, u'NInserts': 147631086, u'DiskUsed': 51589120}, u'NGets': 19497057, u'NReadBytesSet': 88249520622L, u'NReadBytesCompact': 92149564480L, u'IndexResidentRatio': 0.03860214725136757, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.8999999761581421, u'BlockCacheHitRatio': 69.47015380859375, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 87648689900573L, u'BloomFilterMemUsedRatio': 0.2786847949028015, u'NWriteBytes': 319264738923L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.0, u'WriteAmp': 2.5125083923339844, u'NBlocksCached': 27131, u'LogicalDataSize': 39500689267L, u'ActiveBloomFilterMemUsed': 41555408, u'NFileCountCompacts': 19037, u'NTableFiles': 2661, u'FlushQueueSize': 72, u'TableObjectMemUsed': 673320, u'BlockCacheQuota': 84985261, u'TotalDiskUsage': 98418475008L, u'NPurgerRuns': 165, u'ReadIOAmpSet': 0.0, u'NSyncs': 254646, u'NReadIOsSetPerSec': 5.433333396911621, u'NReadIOsPerSec': 40.766666412353516, u'NTablesDeleted': 613192, u'BloomFilterFPR': 0.015387307852506638, u'NBloomFilterHitsPerSec': 1191.6666259765625, u'WriteCacheMemUsed': 2501704, u'RecentBloomFilterFPR': 0.0164791326969862, u'BytesPerRead': 4490957.0, u'BytesIncomingPerSec': 261037.96875, u'NReadIOsGetPerSec': 6.566666603088379, u'NGetStats': 308079, u'TotalIndexBlocksSize': 618981667, u'NGetStatsPerSec': 1.7666666507720947, u'NWriteBytesCompact': 70302490624L, u'NReadIOsSet': 33113979, u'ActiveDataSize': 39500689267L, u'NWriteBytesCompactPerSec': 84241.0703125, u'NBloomFilterMissesPerSec': 19.96666717529297, u'NCommitBatches': 6867145, u'NGetStatsComputed': 308079, u'NWriteIOs': 8167141, u'NReadIOsGet': 32837380537L, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 21246316, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 138.1999969482422, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.6666666865348816, u'ManagedFileObjectMemUsed': 260640, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 83727712, u'NWriterCompacts': 0, u'NReadIOs': 32870916720L, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 1401541222, u'NTablesCreated': 188633, u'ReadIOAmp': 311.9434047997538, u'MemtablesMemUsed': 189464, u'NBloomFilterHits': 974116485, u'TotalIndexBlocksSize': 27358842, u'CheckpointOverhead': 50991104, u'NItemsGet': 105267107, u'NWriteIOs': 375849, u'NReadBytes': 87708985824373L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1972579, u'SizeRatio': 0.042754498395052826, u'NWriteBytes': 6720868352L, u'BloomFilterMemUsed': 26336, u'NumTables': 84, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1855488}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 36664682, u'SizeRatio': 0.7946855805136941, u'NWriteBytes': 23776112640L, u'BloomFilterMemUsed': 1132752, u'NumTables': 62, u'WastedSpace': 76832, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 36687872}, {u'TargetSize': 173786109, u'LogicalSize': 160481440, u'SizeRatio': 0.9234422758150365, u'NWriteBytes': 13263843328L, u'BloomFilterMemUsed': 4881664, u'NumTables': 90, u'WastedSpace': 415749, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 160882688}, {u'TargetSize': 1737861154, u'LogicalSize': 1737861154, u'SizeRatio': 1.0, u'NWriteBytes': 21734068224L, u'BloomFilterMemUsed': 35512992, u'NumTables': 243, u'WastedSpace': 6105123, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1743953920}], u'NItemsGarbageCollected': 38052418, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 26668304, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58560, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 32859841117L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 185385, u'CompactWriter': {u'NDataBlockWrites': 21333237, u'NWriteBytes': 58279616512L, u'NIndexBlockWrites': 275265, u'NBytesIncoming': 81289308126L, u'NSyncs': 0, u'NInserts': 1401541222, u'NWriteIOs': 129500}, u'NWriteBytes': 65136862098L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 32837379771L, u'NReadBytes': 87648689542795L, u'NDataBlockReads': 59051534688L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 115862310708L}, u'NBloomFilterMisses': 15223276, u'NItemsMerged': 0, u'LogicalDiskSize': 1737861154, u'TotalBloomFilterMemUsed': 41558824, u'WriteAmp': 15.498703639684464, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 41553744, u'LogicalDataSize': 1577379714, u'NTableFiles': 610, u'NFileCountCompacts': 9557, u'TableObjectMemUsed': 151280, u'TreeSnapshotMemoryUsed': 95968, u'TotalDiskUsage': 1994731520, u'BytesIncoming': 4202729700L, u'NItemsDel': 0, u'PendingCheckpointBytes': 622592, u'NCompacts': 12683, u'NItemsPut': 85770050, u'NSyncs': 84882, u'CompactReader': {u'NReadIOs': 22461346, u'NReadBytes': 60296281578L, u'NDataBlockReads': 22032921, u'NBytesOutgoing': 0, u'NIndexBlockReads': 428425}, u'ActiveDiskUsage': 1943379968, u'NTablesDeleted': 188154, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015387308384950274, u'Writer': {u'NDataBlockWrites': 1423081, u'NWriteBytes': 6857245586L, u'NIndexBlockWrites': 161862, u'NBytesIncoming': 4974603774L, u'NSyncs': 84882, u'NInserts': 85769078, u'NWriteIOs': 246349}}, u'NReadBytesCompactPerSec': 86144.1328125, u'TableMetaMemUsed': 780151, u'NDeletes': 0, u'BlockCacheHitsPerSec': 452.8666687011719, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 15223276, u'LogicalDiskSize': 74688450769L, u'NSets': 85770050, u'TotalBloomFilterMemUsed': 41560488, u'BufferMemUsed': 494712, u'RecentReadAmp': 0.0, u'NGetsPerSec': 0.0, u'NInserts': 0, u'NCommitBatchesPerSec': 118.06666564941406, u'FragmentationWithOverhead': 0.5986455678939819, u'NCompactsPerSec': 0.23333333432674408, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 104027.6015625, u'TreeSnapshotMemoryUsed': 346304, u'BytesIncoming': 127070115845L, u'NCompacts': 81441, u'NumPersistentFlushes': 82845, u'NWriteIOsPerSec': 122.03333282470703, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 0.0, u'ActiveDiskUsage': 75251200000L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 1.7666666507720947, u'NReadBytesSetPerSec': 14405.0, u'BlockCacheHits': 130745614, u'NWriteBytesPerSec': 789965.1875, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 170652, u'ReadIOAmp': 0.029557030405926842, u'MemtablesMemUsed': 631832, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 11496, u'CheckpointOverhead': 851968, u'NItemsGet': 25916, u'NWriteIOs': 255534, u'NReadBytes': 170005653, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1087299, u'SizeRatio': 0.023566571148959072, u'NWriteBytes': 1320415232, u'BloomFilterMemUsed': 960, u'NumTables': 120, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 983040}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 125103, u'SizeRatio': 0.0027115345001220703, u'NWriteBytes': 77611008, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 41257020, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 6420, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 22560, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 341808, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 67426, u'CompactWriter': {u'NDataBlockWrites': 9474, u'NWriteBytes': 77611008, u'NIndexBlockWrites': 9474, u'NBytesIncoming': 43070231, u'NSyncs': 0, u'NInserts': 502122, u'NWriteIOs': 9474}, u'NWriteBytes': 1441495034, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 766, u'NReadBytes': 357778, u'NDataBlockReads': 651, u'NBytesOutgoing': 0, u'NIndexBlockReads': 651}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 125103, u'TotalBloomFilterMemUsed': 1664, u'WriteAmp': 0.35156013321621465, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1664, u'LogicalDataSize': 125103, u'NTableFiles': 235, u'NFileCountCompacts': 9474, u'TableObjectMemUsed': 58280, u'TreeSnapshotMemoryUsed': 44352, u'TotalDiskUsage': 1925120, u'BytesIncoming': 4100280145L, u'NItemsDel': 0, u'PendingCheckpointBytes': 24576, u'NCompacts': 9474, u'NItemsPut': 41259615, u'NSyncs': 84882, u'CompactReader': {u'NReadIOs': 341042, u'NReadBytes': 169647875, u'NDataBlockReads': 170521, u'NBytesOutgoing': 0, u'NIndexBlockReads': 170521}, u'ActiveDiskUsage': 1073152, u'NTablesDeleted': 170521, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 161178, u'NWriteBytes': 1363884026, u'NIndexBlockWrites': 161178, u'NBytesIncoming': 120591776, u'NSyncs': 84882, u'NInserts': 1578411, u'NWriteIOs': 246060}}}}} 2021-12-15 11:52:59,339 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.4711271822452545, 0.47071683406829834, 0.4717790484428406, 0.4745238125324249, 0.4779290556907654, 0.47413772344589233, 0.46606138348579407, 0.47408050298690796, 0.4731444716453552, 0.47421035170555115, 0.46629855036735535, 0.47243982553482056, 0.46888890862464905, 0.4712626338005066, 0.47220516204833984, 0.4725339710712433, 0.47180166840553284, 0.46983346343040466, 0.4754580557346344, 0.4721551239490509, 0.47678089141845703, 0.47185009717941284, 0.4720655381679535, 0.4641740620136261, 0.472177118062973, 0.4653347432613373, 0.4675668179988861, 0.4707280993461609, 0.4801383912563324, 0.47957515716552734, 0.47644126415252686, 0.4722316265106201, 0.4684500992298126, 0.46264562010765076, 0.47917962074279785, 0.46574145555496216, 0.4679048955440521, 0.46603918075561523, 0.4734269976615906, 0.46631020307540894, 0.4686034321784973, 0.46931368112564087, 0.46123042702674866, 0.47262734174728394, 0.4657122492790222, 0.4707776606082916, 0.47364336252212524, 0.46841126680374146, 0.4716262221336365, 0.46581965684890747, 0.4633273184299469, 0.4628564119338989, 0.47091415524482727, 0.46270671486854553, 0.4755952060222626, 0.47680142521858215, 0.4628461003303528, 0.47016847133636475, 0.4669843912124634, 0.4661525785923004, 0.46300339698791504, 0.4689368009567261, 0.46725574135780334, 0.46607640385627747], '172.23.110.66': [0.4699631333351135, 0.46962788701057434, 0.47678473591804504, 0.46689221262931824, 0.46999526023864746, 0.473696231842041, 0.47548073530197144, 0.47012025117874146, 0.47539567947387695, 0.4748585820198059, 0.47712352871894836, 0.4768355190753937, 0.47731006145477295, 0.47726085782051086, 0.47460830211639404, 0.4780045747756958, 0.47051742672920227, 0.47114163637161255, 0.47345879673957825, 0.47665107250213623, 0.47367435693740845, 0.4751553535461426, 0.47462353110313416, 0.4671732783317566, 0.47425249218940735, 0.4704550802707672, 0.4784339964389801, 0.4770553708076477, 0.4745992124080658, 0.4727526009082794, 0.4775095582008362, 0.47225868701934814, 0.4771101474761963, 0.4709606468677521, 0.47746989130973816, 0.4773801863193512, 0.47635993361473083, 0.47137245535850525, 0.4691702127456665, 0.4722342789173126, 0.4704916775226593, 0.47466203570365906, 0.4729294180870056, 0.47586241364479065, 0.4719077944755554, 0.47133782505989075, 0.4730033576488495, 0.47582194209098816, 0.4755595028400421, 0.47710201144218445, 0.46750032901763916, 0.47266072034835815, 0.47648048400878906, 0.4767158329486847, 0.47260650992393494, 0.4711926281452179, 0.47209954261779785, 0.4695664644241333, 0.4736367464065552, 0.4686281085014343, 0.46835315227508545, 0.4670974314212799, 0.4767880439758301, 0.4750613272190094], '172.23.110.64': [0.4690331816673279, 0.46419015526771545, 0.47548648715019226, 0.4777657985687256, 0.4699418842792511, 0.4700976610183716, 0.46708300709724426, 0.4741297960281372, 0.472805917263031, 0.4722018837928772, 0.4721604287624359, 0.4716688096523285, 0.4701085090637207, 0.47330957651138306, 0.46768009662628174, 0.4752376079559326, 0.4717439115047455, 0.46720415353775024, 0.47010624408721924, 0.4669409990310669, 0.4750244617462158, 0.4663310945034027, 0.46786630153656006, 0.47056373953819275, 0.4692346155643463, 0.4639638662338257, 0.47427213191986084, 0.47215402126312256, 0.4728222191333771, 0.473244845867157, 0.47008800506591797, 0.47910213470458984, 0.4638209342956543, 0.4685410261154175, 0.47310635447502136, 0.4702916443347931, 0.4686886966228485, 0.47572755813598633, 0.4694455564022064, 0.4740130305290222, 0.4730934500694275, 0.47939157485961914, 0.4718114137649536, 0.46698275208473206, 0.4650636911392212, 0.47489646077156067, 0.46910372376441956, 0.4642581343650818, 0.4700714647769928, 0.47192519903182983, 0.4741378724575043, 0.4694833755493164, 0.4743983745574951, 0.4703124165534973, 0.471835732460022, 0.47042426466941833, 0.4713582992553711, 0.4750344157218933, 0.47780224680900574, 0.47236567735671997, 0.465889036655426, 0.4716596305370331, 0.46927762031555176, 0.47223424911499023]} 2021-12-15 11:52:59,701 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 48.0 2021-12-15 11:53:00,069 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 48.0 2021-12-15 11:53:00,433 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 48.0 2021-12-15 11:53:00,434 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 48.0, '172.23.110.66': 48.0, '172.23.110.64': 48.0} 2021-12-15 11:53:00,437 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 11:53:00,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:53:00,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 11:53:00,884 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:53:00,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 11:53:00,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 11:53:00,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 11:53:00,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 11:53:00,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 11:53:00,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 11:53:01,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 11:53:01,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 11:53:01,059 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 11:53:01,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 11:53:01,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:01,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:01,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:01,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:01,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:01,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:01,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:01,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:01,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:01,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:01,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 11:53:01,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 11:53:01,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:01,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:01,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 11:53:01,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 11:53:01,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 11:53:01,790 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 11:53:01,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 11:53:01,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:01,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 11:53:01,855 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 11:53:01,855 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 11:53:01,858 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 11:53:01,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:53:02,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 11:53:02,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:53:02,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 11:53:02,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 11:53:02,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 11:53:02,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 11:53:02,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 11:53:02,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 11:53:02,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 11:53:02,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 11:53:02,548 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 11:53:02,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 11:53:02,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:02,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:02,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:02,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:02,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:02,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:02,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:02,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:02,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:02,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:02,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 11:53:02,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 11:53:03,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 11:53:03,015 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 11:53:03,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 11:53:03,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:03,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 11:53:03,059 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 11:53:03,059 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 11:53:03,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 11:53:03,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 11:53:03,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 11:53:03,572 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 11:53:03,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 11:53:03,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 11:53:03,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 11:53:03,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 11:53:03,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 11:53:03,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 11:53:03,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 11:53:03,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 11:53:03,717 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 11:53:03,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 11:53:03,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:03,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:03,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:03,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 11:53:03,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:03,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 11:53:04,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:04,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 11:53:04,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 11:53:04,127 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 11:53:04,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 11:53:04,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 11:53:04,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 11:53:04,190 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 11:53:04,190 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 ############################################################ # # RAM FootPrint: 6861.31640625 # ############################################################ Read Start: 0 Read End: 40000000 Update Start: 80000000 Update End: 800000000 Expiry Start: 0 Expiry End: 0 Delete Start: 40000000 Delete End: 80000000 Create Start: 80000000 Create End: 800000000 Final Start: 80000000 Final End: 800000000 2021-12-15 11:53:04,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:04,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection46_10_1639597985.24 2021-12-15 11:53:05,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:05,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection47_9_1639597986.25 2021-12-15 11:53:06,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:06,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection44_8_1639597987.26 2021-12-15 11:53:07,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:07,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection45_7_1639597988.28 2021-12-15 11:53:08,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:08,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection48_6_1639597989.29 2021-12-15 11:53:09,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:09,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection49_5_1639597990.43 2021-12-15 11:53:10,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:10,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection42_4_1639597991.45 2021-12-15 11:53:11,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:11,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection43_3_1639597992.46 2021-12-15 11:53:12,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:12,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection40_2_1639597993.47 2021-12-15 11:53:13,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:13,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Starting Loader_GleamBookUsers0__default_VolumeCollection41_1_1639597994.49 2021-12-15 11:53:14,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:14,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:15,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:15,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:16,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:16,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:17,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:17,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:18,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:18,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:19,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:19,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:20,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:20,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:21,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:21,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:22,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:22,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:23,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:23,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:24,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:24,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:25,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:25,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:26,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:26,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:27,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:27,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:28,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:28,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:29,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:29,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:30,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:30,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:31,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:31,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:32,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:32,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:33,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:33,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:34,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:34,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:35,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:35,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:36,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:36,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:37,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:37,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:38,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:38,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:39,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:39,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:40,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:40,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:41,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:41,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:42,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:42,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:43,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:43,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:44,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:44,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:45,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:45,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:46,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:46,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:47,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:47,857 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:48,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:48,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:49,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:49,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:50,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:50,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:51,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:51,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:52,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:52,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-12-15 11:53:53,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> perform_load():L577 2021-12-15 11:53:53,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 1 seconds. Reason: None ############################################################ # # Step 4: Rebalance in with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.68'] Servers coming in : ['172.23.110.68'] with services: ['kv'] Servers going out : [] NEW AVAILABLE nodes for cluster: ['172.23.110.67'] 2021-12-15 11:53:54,944 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.68]_OUT=[]_1639598034.91 2021-12-15 11:53:54,944 | infra | INFO | pool-3-thread-20 | [task:start_task:90] Thread 'Rebalance_task_IN=[172.23.110.68]_OUT=[]_1639598034.91' started 2021-12-15 11:53:54,947 | test | DEBUG | pool-3-thread-20 | [rest_client:add_node:1184] Adding remote node 172.23.110.68:8091 to cluster 172.23.110.64:8091 2021-12-15 11:53:54,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L469 2021-12-15 11:53:54,950 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-15 11:53:58,780 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-15 11:53:58,835 | test | DEBUG | pool-3-thread-20 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 11:53:58,835 | test | INFO | pool-3-thread-20 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 12.3688376459 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 11.8857511066 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 12.3827129373 | Cluster node | | 172.23.110.68 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 11:53:58,844 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:53:58,844 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:53:58,855 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:53:58,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:53:58,859 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:08,872 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:08,872 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:08,888 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:08,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:08,891 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:18,905 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:18,905 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:18,924 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:18,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:18,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:28,941 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:28,941 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:28,961 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:28,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:28,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:38,979 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:38,979 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:38,999 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:39,000 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:39,000 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:49,016 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:49,016 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:49,045 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:49,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:49,046 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:54:54,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-15 11:54:54,957 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-15 11:54:59,062 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:54:59,062 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:54:59,082 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:54:59,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:54:59,085 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:09,099 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:55:09,101 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:55:09,121 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:09,122 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:09,124 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:19,138 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:55:19,138 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 11:55:19,157 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:19,160 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:19,160 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:24,984 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 11:55:24,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:55:24,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:29,174 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 11:55:29,174 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 11:55:29,190 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:29,193 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:29,193 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:35,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 11:55:35,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:55:35,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:39,209 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 11:55:39,210 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 11:55:39,230 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:39,233 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:39,233 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:45,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 11:55:45,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:55:45,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:49,250 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 11:55:49,250 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 11:55:49,270 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:49,273 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:49,273 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:55,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:55:55,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:55:55,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:55:59,289 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:55:59,289 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:55:59,308 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:55:59,312 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:55:59,312 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:05,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:05,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:05,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:09,326 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:09,326 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:56:09,348 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:09,351 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:09,351 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:15,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:15,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:15,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:19,368 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:19,368 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:56:19,390 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:19,392 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:19,392 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:25,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:25,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:25,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:29,410 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:29,411 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:56:29,430 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:29,433 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:29,434 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:35,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:35,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:35,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:39,450 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:39,450 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:56:39,469 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:39,470 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:39,471 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:45,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:45,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:45,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:49,487 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 11:56:49,487 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 11:56:49,503 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:49,523 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:49,523 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:55,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.98 % 2021-12-15 11:56:55,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:56:55,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:56:59,536 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 11:56:59,536 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 11:56:59,552 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:56:59,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:56:59,555 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:05,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 11:57:05,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:05,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:09,569 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 11:57:09,569 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 11:57:09,585 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:09,588 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:09,588 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:15,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 11:57:15,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:15,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:19,602 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:19,602 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:57:19,618 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:19,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:19,621 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:25,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:25,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:25,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:29,635 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:29,635 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:57:29,653 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:29,655 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:29,655 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:35,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:35,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:35,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:39,684 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:39,686 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:57:39,701 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:39,704 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:39,704 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:45,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:45,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:45,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:49,717 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:49,717 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:57:49,746 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:49,747 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:49,749 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:55,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:55,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:57:55,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:57:59,759 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:57:59,760 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:57:59,776 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:57:59,778 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:57:59,778 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:05,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:58:05,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:05,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:09,789 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 11:58:09,790 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 11:58:09,809 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:09,811 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:09,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:15,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 11:58:15,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:15,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:19,823 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 11:58:19,823 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 11:58:19,838 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:19,839 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:19,841 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:25,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:25,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:25,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:29,851 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:29,851 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 11:58:29,868 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:29,869 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:29,869 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:35,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:35,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:35,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:39,881 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:39,882 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 11:58:39,900 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:39,901 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:39,901 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:45,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:45,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:45,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:49,913 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 11:58:49,913 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 11:58:49,927 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:49,928 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:49,928 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:55,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:58:55,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:58:55,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:58:59,943 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:58:59,943 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 11:58:59,966 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:58:59,969 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:58:59,969 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:05,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:05,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:05,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:09,982 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:09,982 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 11:59:09,996 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:59:09,997 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:59:09,999 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:15,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:15,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:15,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:20,010 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:20,012 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 11:59:20,026 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:59:20,028 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:59:20,028 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:25,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:25,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:25,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:30,042 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:30,042 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 11:59:30,062 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:59:30,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:59:30,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:35,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:35,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:35,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:40,079 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 11:59:40,079 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 11:59:40,095 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:59:40,098 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:59:40,098 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:45,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-15 11:59:45,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:45,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:50,112 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.74 % 2021-12-15 11:59:50,114 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.74 2021-12-15 11:59:50,138 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 11:59:50,140 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 11:59:50,140 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 11:59:55,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.74 % 2021-12-15 11:59:55,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 11:59:55,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:00,160 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.74 % 2021-12-15 12:00:00,160 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.74 2021-12-15 12:00:00,184 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:00,187 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:00,187 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:05,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.74 % 2021-12-15 12:00:05,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:05,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:10,200 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 12:00:10,200 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.93 2021-12-15 12:00:10,217 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:10,219 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:10,219 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:15,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 12:00:15,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:15,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:20,230 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 12:00:20,232 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 2.93 2021-12-15 12:00:20,244 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:20,246 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:20,247 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:25,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 12:00:25,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:25,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:30,260 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 12:00:30,262 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 12:00:30,282 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:30,283 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:30,283 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:35,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 12:00:35,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:35,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:40,298 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 12:00:40,299 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 12:00:40,313 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:40,315 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:40,316 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:45,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:00:45,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:45,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:50,332 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:00:50,332 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.32 2021-12-15 12:00:50,352 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:00:50,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:00:50,357 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:00:55,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:00:55,767 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:00:55,767 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:00,368 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:01:00,369 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.32 2021-12-15 12:01:00,388 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:00,391 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:00,391 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:05,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:01:05,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:05,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:10,407 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 12:01:10,407 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.32 2021-12-15 12:01:10,427 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:10,430 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:10,431 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:15,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 12:01:15,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:15,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:20,447 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 12:01:20,447 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 12:01:20,467 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:20,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:20,473 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:25,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 12:01:25,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:25,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:30,486 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 12:01:30,486 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 12:01:30,502 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:30,503 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:30,503 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:35,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 12:01:35,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:35,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:40,516 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 12:01:40,516 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 12:01:40,530 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:40,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:40,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:45,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 12:01:45,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:45,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:50,546 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 12:01:50,546 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 12:01:50,562 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:01:50,565 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:01:50,565 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:01:55,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.91 % 2021-12-15 12:01:55,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:01:55,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:00,582 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.91 % 2021-12-15 12:02:00,582 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 3.91 2021-12-15 12:02:00,598 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:00,601 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:00,601 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:05,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.91 % 2021-12-15 12:02:05,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:05,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:10,615 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:10,615 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-15 12:02:10,628 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:10,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:10,631 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:15,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:15,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:15,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:20,644 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:20,644 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-15 12:02:20,671 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:20,674 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:20,674 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:25,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:25,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:25,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:30,687 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:30,687 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-15 12:02:30,706 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:30,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:30,709 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:35,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:35,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:35,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:40,723 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 12:02:40,723 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-15 12:02:40,744 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:40,763 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:40,763 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:45,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 12:02:45,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:45,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:50,778 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 12:02:50,779 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 12:02:50,798 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:02:50,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:02:50,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:02:56,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 12:02:56,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:02:56,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:00,813 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-15 12:03:00,815 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.49 2021-12-15 12:03:00,832 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:00,835 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:00,835 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:06,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-15 12:03:06,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:06,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:10,861 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-15 12:03:10,861 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.49 2021-12-15 12:03:10,878 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:10,881 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:10,881 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:16,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-15 12:03:16,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:16,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:20,892 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 12:03:20,892 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-15 12:03:20,907 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:20,910 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:20,910 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:26,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 12:03:26,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:26,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:30,921 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 12:03:30,921 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-15 12:03:30,937 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:30,940 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:30,940 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:36,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 12:03:36,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:36,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:40,953 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.88 % 2021-12-15 12:03:40,953 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.88 2021-12-15 12:03:40,967 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:40,969 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:40,969 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:46,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.88 % 2021-12-15 12:03:46,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:46,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:50,983 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.88 % 2021-12-15 12:03:50,983 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.88 2021-12-15 12:03:51,000 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:03:51,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:03:51,003 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:03:56,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.88 % 2021-12-15 12:03:56,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:03:56,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:01,015 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.88 % 2021-12-15 12:04:01,015 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 4.88 2021-12-15 12:04:01,035 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:01,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:01,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:06,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 12:04:06,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:06,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:11,048 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 12:04:11,049 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.08 2021-12-15 12:04:11,062 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:11,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:11,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:16,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 12:04:16,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:16,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:21,076 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 12:04:21,078 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 12:04:21,092 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:21,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:21,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:26,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 12:04:26,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:26,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:31,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 12:04:31,108 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 12:04:31,127 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:31,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:31,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:36,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 12:04:36,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:36,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:41,142 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 12:04:41,142 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 12:04:41,160 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:41,161 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:41,161 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:46,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 12:04:46,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:46,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:51,176 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 12:04:51,177 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.47 2021-12-15 12:04:51,194 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:04:51,196 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:04:51,196 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:04:56,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 12:04:56,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:04:56,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:01,207 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 12:05:01,207 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 12:05:01,220 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:01,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:01,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:06,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 12:05:06,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:06,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:11,236 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 12:05:11,236 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 12:05:11,259 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:11,260 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:11,260 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:16,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 12:05:16,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:16,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:21,276 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 12:05:21,276 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 12:05:21,295 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:21,296 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:21,296 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:26,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.86 % 2021-12-15 12:05:26,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:26,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:31,309 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.86 % 2021-12-15 12:05:31,309 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 5.86 2021-12-15 12:05:31,323 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:31,326 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:31,326 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:36,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:05:36,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:36,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:41,338 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:05:41,338 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 12:05:41,352 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:41,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:41,355 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:46,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:05:46,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:46,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:51,371 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:05:51,371 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 12:05:51,388 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:05:51,391 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:05:51,391 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:05:56,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:05:56,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:05:56,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:01,404 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:06:01,404 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 12:06:01,426 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:01,428 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:01,430 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:06,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:06:06,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:06,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:11,444 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 12:06:11,444 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 12:06:11,461 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:11,464 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:11,464 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:16,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.25 % 2021-12-15 12:06:16,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:16,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:21,477 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.25 % 2021-12-15 12:06:21,479 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.25 2021-12-15 12:06:21,496 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:21,499 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:21,499 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:26,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.25 % 2021-12-15 12:06:26,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:26,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:31,513 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.25 % 2021-12-15 12:06:31,513 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.25 2021-12-15 12:06:31,529 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:31,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:31,532 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:36,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 12:06:36,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:36,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:41,545 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 12:06:41,546 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.45 2021-12-15 12:06:41,559 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:41,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:41,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:46,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-15 12:06:46,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:46,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:51,575 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:06:51,575 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 12:06:51,592 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:06:51,594 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:06:51,594 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:06:56,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:06:56,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:06:56,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:01,607 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:01,607 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 12:07:01,624 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:01,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:01,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:06,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:06,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:06,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:11,640 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:11,641 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 12:07:11,657 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:11,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:11,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:16,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:16,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:16,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:21,671 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:21,673 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 12:07:21,690 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:21,693 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:21,693 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:26,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:26,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:26,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:31,710 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:31,710 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 12:07:31,732 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:31,734 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:31,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:36,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 12:07:36,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:36,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:41,750 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-15 12:07:41,750 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-15 12:07:41,766 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:41,770 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:41,770 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:46,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-15 12:07:46,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:46,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:51,783 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-15 12:07:51,783 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-15 12:07:51,802 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:07:51,805 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:07:51,805 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:07:56,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-15 12:07:56,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:07:56,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:01,819 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 12:08:01,819 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-15 12:08:01,835 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:01,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:01,838 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:06,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.42 % 2021-12-15 12:08:06,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:06,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:11,854 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.42 % 2021-12-15 12:08:11,854 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.42 2021-12-15 12:08:11,869 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:11,872 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:11,872 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:16,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.42 % 2021-12-15 12:08:16,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:16,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:21,888 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:21,888 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:08:21,904 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:21,910 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:21,910 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:26,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:26,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:26,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:31,921 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:31,921 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:08:31,938 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:31,941 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:31,941 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:36,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:36,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:36,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:41,953 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:41,953 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:08:41,970 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:41,971 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:41,971 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:46,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:46,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:46,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:51,983 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:51,983 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:08:52,006 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:08:52,009 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:08:52,009 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:08:56,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:08:56,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:08:56,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:02,020 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:09:02,020 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:09:02,035 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:02,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:02,036 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:06,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 12:09:06,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:06,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:12,048 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 12:09:12,048 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 7.81 2021-12-15 12:09:12,062 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:12,065 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:12,065 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:16,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.01 % 2021-12-15 12:09:16,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:16,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:22,078 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.01 % 2021-12-15 12:09:22,078 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.01 2021-12-15 12:09:22,098 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:22,101 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:22,101 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:26,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.20 % 2021-12-15 12:09:26,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:26,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:32,111 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.20 % 2021-12-15 12:09:32,111 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.2 2021-12-15 12:09:32,128 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:32,130 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:32,130 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:36,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.20 % 2021-12-15 12:09:36,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:36,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:42,142 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.20 % 2021-12-15 12:09:42,144 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.2 2021-12-15 12:09:42,164 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:42,167 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:42,167 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:46,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:09:46,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:46,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:52,177 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:09:52,178 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.4 2021-12-15 12:09:52,191 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:09:52,194 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:09:52,194 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:09:56,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:09:56,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:09:56,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:02,209 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:10:02,209 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.4 2021-12-15 12:10:02,226 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:02,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:02,230 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:06,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:10:06,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:06,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:12,246 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:10:12,246 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.4 2021-12-15 12:10:12,267 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:12,269 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:12,269 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:16,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:10:17,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:17,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:22,282 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.40 % 2021-12-15 12:10:22,282 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.4 2021-12-15 12:10:22,298 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:22,301 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:22,301 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:27,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 12:10:27,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:27,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:32,315 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 12:10:32,315 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 12:10:32,339 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:32,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:32,342 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:37,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.79 % 2021-12-15 12:10:37,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:37,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:42,355 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.79 % 2021-12-15 12:10:42,355 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.79 2021-12-15 12:10:42,372 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:42,377 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:42,377 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:47,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.79 % 2021-12-15 12:10:47,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:47,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:52,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.79 % 2021-12-15 12:10:52,390 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.79 2021-12-15 12:10:52,410 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:10:52,414 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:10:52,414 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:10:57,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 12:10:57,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:10:57,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:02,430 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 12:11:02,430 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.99 2021-12-15 12:11:02,453 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:02,457 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:02,457 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:07,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 12:11:07,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:07,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:12,473 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 12:11:12,473 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 8.99 2021-12-15 12:11:12,503 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:12,506 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:12,506 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:17,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.18 % 2021-12-15 12:11:17,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:17,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:22,523 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.18 % 2021-12-15 12:11:22,523 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.18 2021-12-15 12:11:22,540 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:22,543 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:22,545 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:27,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.18 % 2021-12-15 12:11:27,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:27,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:32,565 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.18 % 2021-12-15 12:11:32,566 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.18 2021-12-15 12:11:32,582 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:32,584 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:32,584 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:37,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.18 % 2021-12-15 12:11:37,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:37,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:42,595 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 12:11:42,595 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.38 2021-12-15 12:11:42,609 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:42,611 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:42,611 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:47,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 12:11:47,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:47,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:52,630 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 12:11:52,630 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.38 2021-12-15 12:11:52,648 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:11:52,651 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:11:52,651 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:11:57,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 12:11:57,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:11:57,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:02,664 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.57 % 2021-12-15 12:12:02,664 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.57 2021-12-15 12:12:02,680 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:02,680 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:02,681 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:07,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.57 % 2021-12-15 12:12:07,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:07,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:12,694 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.57 % 2021-12-15 12:12:12,694 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.57 2021-12-15 12:12:12,709 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:12,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:12,711 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:17,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.57 % 2021-12-15 12:12:17,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:17,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:22,723 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.77 % 2021-12-15 12:12:22,723 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.77 2021-12-15 12:12:22,740 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:22,743 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:22,743 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:27,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.77 % 2021-12-15 12:12:27,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:27,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:32,755 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.77 % 2021-12-15 12:12:32,755 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.77 2021-12-15 12:12:32,773 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:32,776 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:32,776 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:37,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.77 % 2021-12-15 12:12:37,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:37,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:42,790 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.77 % 2021-12-15 12:12:42,790 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.77 2021-12-15 12:12:42,809 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:42,812 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:42,813 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:47,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.96 % 2021-12-15 12:12:47,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:47,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:52,825 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.96 % 2021-12-15 12:12:52,825 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 9.96 2021-12-15 12:12:52,842 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:12:52,845 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:12:52,845 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:12:57,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.96 % 2021-12-15 12:12:57,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:12:57,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:02,861 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.16 % 2021-12-15 12:13:02,861 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.16 2021-12-15 12:13:02,885 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:02,888 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:02,888 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:07,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.16 % 2021-12-15 12:13:07,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:07,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:12,901 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:12,901 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.35 2021-12-15 12:13:12,917 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:12,918 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:12,920 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:17,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:17,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:17,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:22,933 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:22,934 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.35 2021-12-15 12:13:22,953 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:22,954 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:22,956 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:27,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:27,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:27,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:32,966 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:32,967 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.35 2021-12-15 12:13:32,980 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:32,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:32,983 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:37,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:37,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:37,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:43,000 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.35 % 2021-12-15 12:13:43,002 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.35 2021-12-15 12:13:43,026 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:43,029 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:43,029 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:47,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.55 % 2021-12-15 12:13:47,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:47,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:53,042 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.55 % 2021-12-15 12:13:53,042 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.55 2021-12-15 12:13:53,059 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:13:53,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:13:53,063 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:13:57,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.55 % 2021-12-15 12:13:57,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:13:57,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:03,076 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.55 % 2021-12-15 12:14:03,076 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.55 2021-12-15 12:14:03,092 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:03,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:03,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:07,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.55 % 2021-12-15 12:14:07,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:07,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:13,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.74 % 2021-12-15 12:14:13,108 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.74 2021-12-15 12:14:13,122 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:13,125 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:13,125 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:17,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.94 % 2021-12-15 12:14:17,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:17,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:23,140 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.94 % 2021-12-15 12:14:23,140 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 10.94 2021-12-15 12:14:23,158 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:23,161 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:23,161 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:27,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.94 % 2021-12-15 12:14:27,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:27,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:33,171 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:33,173 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.14 2021-12-15 12:14:33,187 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:33,188 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:33,190 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:37,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:37,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:37,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:43,203 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:43,203 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.14 2021-12-15 12:14:43,220 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:43,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:43,223 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:47,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:47,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:47,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:53,236 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:53,236 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.14 2021-12-15 12:14:53,256 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:14:53,260 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:14:53,260 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:14:57,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:14:57,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:14:57,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:03,273 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:15:03,273 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.14 2021-12-15 12:15:03,289 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:03,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:03,293 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:07,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:15:07,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:07,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:13,305 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.14 % 2021-12-15 12:15:13,305 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.14 2021-12-15 12:15:13,321 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:13,323 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:13,323 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:17,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-15 12:15:17,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:17,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:23,334 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-15 12:15:23,334 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-15 12:15:23,348 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:23,349 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:23,349 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:27,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-15 12:15:27,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:27,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:33,361 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-15 12:15:33,361 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-15 12:15:33,377 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:33,381 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:33,381 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:37,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-15 12:15:37,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:37,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:43,395 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 12:15:43,395 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.53 2021-12-15 12:15:44,417 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:44,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:44,421 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:47,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.72 % 2021-12-15 12:15:47,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:47,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:54,433 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.72 % 2021-12-15 12:15:54,433 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.72 2021-12-15 12:15:54,450 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:15:54,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:15:54,453 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:15:57,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:15:57,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:15:57,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:04,467 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:04,467 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.92 2021-12-15 12:16:04,483 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:04,487 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:04,487 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:07,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:07,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:07,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:14,499 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:14,499 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.92 2021-12-15 12:16:14,516 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:14,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:14,519 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:17,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:17,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:17,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:24,529 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:24,529 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.92 2021-12-15 12:16:24,546 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:24,548 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:24,548 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:27,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:27,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:27,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:34,558 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:34,559 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 11.92 2021-12-15 12:16:34,573 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:34,575 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:34,576 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:37,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.92 % 2021-12-15 12:16:37,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:37,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:44,586 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.11 % 2021-12-15 12:16:44,588 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.11 2021-12-15 12:16:44,601 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:44,602 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:44,602 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:47,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.31 % 2021-12-15 12:16:47,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:47,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:54,614 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.31 % 2021-12-15 12:16:54,614 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.31 2021-12-15 12:16:54,628 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:16:54,630 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:16:54,630 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:16:57,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.31 % 2021-12-15 12:16:57,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:16:57,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:04,642 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.31 % 2021-12-15 12:17:04,642 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.31 2021-12-15 12:17:04,657 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:04,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:04,660 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:07,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.31 % 2021-12-15 12:17:07,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:07,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:14,671 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 12:17:14,671 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.5 2021-12-15 12:17:14,684 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:14,687 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:14,687 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:17,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 12:17:17,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:17,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:24,707 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 12:17:24,707 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.5 2021-12-15 12:17:24,730 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:24,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:24,736 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:27,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 12:17:27,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:27,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:34,750 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.70 % 2021-12-15 12:17:34,750 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.7 2021-12-15 12:17:34,766 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:34,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:34,769 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:37,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.70 % 2021-12-15 12:17:37,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:37,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:44,785 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.70 % 2021-12-15 12:17:44,786 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.7 2021-12-15 12:17:44,801 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:44,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:44,802 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:47,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.70 % 2021-12-15 12:17:47,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:47,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:54,813 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 12:17:54,813 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 12:17:54,828 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:17:54,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:17:54,829 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:17:57,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 12:17:57,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:17:57,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:04,839 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 12:18:04,839 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 12:18:04,855 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:04,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:04,857 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:07,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:07,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:07,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:14,868 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:14,868 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.09 2021-12-15 12:18:14,887 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:14,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:14,890 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:17,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:17,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:17,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:24,903 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:24,903 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.09 2021-12-15 12:18:24,924 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:24,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:24,927 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:27,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:27,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:27,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:34,940 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:34,940 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.09 2021-12-15 12:18:34,960 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:34,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:34,963 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:37,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.09 % 2021-12-15 12:18:37,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:37,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:44,976 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-15 12:18:44,976 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.28 2021-12-15 12:18:44,992 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:44,994 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:44,994 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:47,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-15 12:18:47,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:47,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:55,006 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-15 12:18:55,006 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.28 2021-12-15 12:18:55,023 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:18:55,025 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:18:55,026 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:18:58,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-15 12:18:58,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:18:58,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:05,036 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.48 % 2021-12-15 12:19:05,036 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.48 2021-12-15 12:19:05,055 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:05,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:05,058 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:08,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.48 % 2021-12-15 12:19:08,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:08,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:15,069 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.48 % 2021-12-15 12:19:15,069 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.48 2021-12-15 12:19:15,088 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:15,089 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:15,089 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:18,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.48 % 2021-12-15 12:19:18,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:18,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:25,104 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.48 % 2021-12-15 12:19:25,105 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.48 2021-12-15 12:19:25,125 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:25,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:25,128 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:28,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.87 % 2021-12-15 12:19:28,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:28,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:35,141 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.87 % 2021-12-15 12:19:35,141 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 13.87 2021-12-15 12:19:35,160 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:35,163 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:35,163 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:38,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.87 % 2021-12-15 12:19:38,081 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:38,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:45,176 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:19:45,177 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:19:45,194 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:45,196 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:45,196 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:48,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:19:48,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:48,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:55,207 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:19:55,207 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:19:55,223 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:19:55,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:19:55,226 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:19:58,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:19:58,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:19:58,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:05,240 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:05,240 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:20:05,259 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:05,262 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:05,263 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:08,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:08,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:08,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:15,273 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:15,273 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:20:15,289 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:15,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:15,292 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:18,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:18,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:18,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:25,303 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:25,303 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:20:25,319 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:25,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:25,322 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:28,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:28,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:28,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:35,334 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 12:20:35,335 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 12:20:35,351 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:35,354 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:35,354 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:38,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.26 % 2021-12-15 12:20:38,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:38,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:45,365 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.46 % 2021-12-15 12:20:45,365 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.46 2021-12-15 12:20:45,381 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:45,384 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:45,385 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:48,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.46 % 2021-12-15 12:20:48,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:48,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:55,397 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.65 % 2021-12-15 12:20:55,397 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.65 2021-12-15 12:20:55,413 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:20:55,414 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:20:55,415 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:20:58,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.65 % 2021-12-15 12:20:58,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:20:58,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:05,437 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.65 % 2021-12-15 12:21:05,437 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.65 2021-12-15 12:21:05,457 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:05,460 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:05,461 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:08,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.65 % 2021-12-15 12:21:08,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:08,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:15,473 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.65 % 2021-12-15 12:21:15,473 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.65 2021-12-15 12:21:15,492 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:15,494 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:15,494 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:18,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.85 % 2021-12-15 12:21:18,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:18,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:25,509 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.85 % 2021-12-15 12:21:25,509 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.85 2021-12-15 12:21:25,525 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:25,528 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:25,528 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:28,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.85 % 2021-12-15 12:21:28,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:28,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:35,539 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.85 % 2021-12-15 12:21:35,539 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 14.85 2021-12-15 12:21:35,559 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:35,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:35,562 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:38,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.85 % 2021-12-15 12:21:38,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:38,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:45,576 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.04 % 2021-12-15 12:21:45,578 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.04 2021-12-15 12:21:45,598 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:45,604 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:45,605 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:48,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.04 % 2021-12-15 12:21:48,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:48,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:55,618 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.04 % 2021-12-15 12:21:55,618 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.04 2021-12-15 12:21:55,635 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:21:55,640 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:21:55,640 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:21:58,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.04 % 2021-12-15 12:21:58,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:21:58,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:05,653 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-15 12:22:05,653 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.24 2021-12-15 12:22:05,667 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:05,670 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:05,671 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:08,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-15 12:22:08,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:08,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:15,680 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-15 12:22:15,680 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.24 2021-12-15 12:22:15,694 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:15,697 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:15,697 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:18,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-15 12:22:18,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:18,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:25,707 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.43 % 2021-12-15 12:22:25,709 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.43 2021-12-15 12:22:25,723 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:25,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:25,726 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:28,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.43 % 2021-12-15 12:22:28,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:28,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:35,736 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.43 % 2021-12-15 12:22:35,736 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.43 2021-12-15 12:22:35,752 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:35,755 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:35,755 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:38,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.43 % 2021-12-15 12:22:38,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:38,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:45,766 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-15 12:22:45,766 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.63 2021-12-15 12:22:45,782 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:45,785 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:45,785 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:48,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-15 12:22:48,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:48,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:55,799 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:22:55,801 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.82 2021-12-15 12:22:55,815 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:22:55,816 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:22:55,816 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:22:58,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:22:58,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:22:58,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:05,828 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:23:05,828 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.82 2021-12-15 12:23:05,842 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:05,844 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:05,844 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:08,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:23:08,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:08,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:15,855 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:23:15,855 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 15.82 2021-12-15 12:23:15,871 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:15,874 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:15,874 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:18,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.82 % 2021-12-15 12:23:18,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:18,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:25,884 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.02 % 2021-12-15 12:23:25,885 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.02 2021-12-15 12:23:25,901 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:25,903 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:25,903 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:28,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.02 % 2021-12-15 12:23:28,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:28,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:35,923 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.02 % 2021-12-15 12:23:35,923 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.02 2021-12-15 12:23:35,938 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:35,940 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:35,940 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:38,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.02 % 2021-12-15 12:23:38,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:38,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:45,950 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:23:45,950 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.21 2021-12-15 12:23:45,966 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:45,967 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:45,969 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:48,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:23:48,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:48,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:55,982 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:23:55,982 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.21 2021-12-15 12:23:56,000 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:23:56,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:23:56,005 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:23:58,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:23:58,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:23:58,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:06,019 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:24:06,019 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.21 2021-12-15 12:24:06,038 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:06,042 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:06,042 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:08,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:24:08,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:08,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:16,053 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:24:16,053 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.21 2021-12-15 12:24:16,072 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:16,075 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:16,075 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:18,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:24:18,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:18,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:26,085 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.21 % 2021-12-15 12:24:26,086 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.21 2021-12-15 12:24:26,102 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:26,105 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:26,105 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:28,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.41 % 2021-12-15 12:24:28,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:28,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:36,118 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:36,118 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.6 2021-12-15 12:24:36,137 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:36,138 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:36,140 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:38,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:38,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:38,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:46,151 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:46,151 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.6 2021-12-15 12:24:46,171 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:46,174 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:46,174 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:48,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:48,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:48,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:56,186 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:56,186 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.6 2021-12-15 12:24:56,207 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:24:56,210 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:24:56,210 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:24:58,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.60 % 2021-12-15 12:24:58,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:24:58,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:06,223 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:06,223 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 12:25:06,240 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:06,243 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:08,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:08,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:08,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:16,256 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:16,256 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 12:25:16,273 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:16,276 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:16,278 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:18,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:18,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:18,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:26,293 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:26,293 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 12:25:26,309 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:26,312 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:26,312 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:28,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:28,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:28,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:36,323 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:36,325 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 12:25:36,345 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:36,348 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:36,348 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:38,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:38,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:38,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:46,359 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:46,359 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 12:25:46,375 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:46,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:46,378 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:48,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 12:25:48,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:48,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:56,390 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.19 % 2021-12-15 12:25:56,390 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.19 2021-12-15 12:25:56,404 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:25:56,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:25:56,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:25:58,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.19 % 2021-12-15 12:25:58,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:25:58,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:06,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.19 % 2021-12-15 12:26:06,418 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.19 2021-12-15 12:26:06,434 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:06,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:06,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:08,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.19 % 2021-12-15 12:26:08,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:08,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:16,448 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.39 % 2021-12-15 12:26:16,448 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.39 2021-12-15 12:26:16,463 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:16,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:16,467 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:18,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.39 % 2021-12-15 12:26:18,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:18,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:26,480 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:26,480 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-15 12:26:26,496 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:26,499 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:26,499 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:28,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:28,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:28,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:36,509 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:36,509 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-15 12:26:36,526 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:36,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:36,529 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:38,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:38,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:38,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:46,542 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:46,542 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-15 12:26:46,558 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:46,559 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:46,559 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:48,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-15 12:26:49,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:49,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:56,572 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:26:56,572 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.78 2021-12-15 12:26:56,591 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:26:56,594 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:26:56,595 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:26:59,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:26:59,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:26:59,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:06,605 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:27:06,605 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.78 2021-12-15 12:27:06,621 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:06,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:06,625 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:09,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:27:09,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:09,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:16,638 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:27:16,640 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.78 2021-12-15 12:27:16,657 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:16,661 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:16,661 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:19,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.78 % 2021-12-15 12:27:19,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:19,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:26,673 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-15 12:27:26,673 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.97 2021-12-15 12:27:26,688 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:26,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:26,691 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:29,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-15 12:27:29,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:29,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:36,703 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-15 12:27:36,703 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 17.97 2021-12-15 12:27:36,720 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:36,723 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:36,723 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:39,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-15 12:27:39,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:39,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:46,734 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 12:27:46,734 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 12:27:46,750 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:46,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:46,753 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:49,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 12:27:49,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:49,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:56,763 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 12:27:56,763 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 12:27:56,778 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:27:56,780 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:27:56,780 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:27:59,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.36 % 2021-12-15 12:27:59,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:27:59,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:06,802 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.36 % 2021-12-15 12:28:06,802 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.36 2021-12-15 12:28:06,819 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:06,822 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:06,822 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:09,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.36 % 2021-12-15 12:28:09,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:09,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:16,834 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:16,834 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.56 2021-12-15 12:28:16,848 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:16,851 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:16,851 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:19,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:19,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:19,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:26,862 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:26,862 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.56 2021-12-15 12:28:26,878 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:26,880 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:26,880 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:29,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:29,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:29,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:36,891 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:36,891 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.56 2021-12-15 12:28:36,905 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:36,907 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:36,907 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:39,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.56 % 2021-12-15 12:28:39,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:39,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:46,920 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:28:46,921 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.75 2021-12-15 12:28:46,937 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:46,938 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:46,940 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:49,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:28:49,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:49,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:56,953 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:28:56,953 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.75 2021-12-15 12:28:56,969 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:28:56,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:28:56,970 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:28:59,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:28:59,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:28:59,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:06,982 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:29:06,983 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.75 2021-12-15 12:29:07,000 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:07,002 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:07,002 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:09,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.75 % 2021-12-15 12:29:09,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:09,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:17,013 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.95 % 2021-12-15 12:29:17,015 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 18.95 2021-12-15 12:29:17,029 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:17,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:17,032 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:19,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.95 % 2021-12-15 12:29:19,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:19,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:27,046 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.14 % 2021-12-15 12:29:27,046 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.14 2021-12-15 12:29:27,062 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:27,065 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:27,065 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:29,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.14 % 2021-12-15 12:29:29,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:29,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:37,075 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.14 % 2021-12-15 12:29:37,076 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.14 2021-12-15 12:29:37,092 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:37,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:37,095 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:39,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.14 % 2021-12-15 12:29:39,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:39,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:47,108 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.34 % 2021-12-15 12:29:47,108 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.34 2021-12-15 12:29:47,122 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:47,125 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:47,125 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:49,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.34 % 2021-12-15 12:29:49,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:49,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:57,135 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.34 % 2021-12-15 12:29:57,135 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.34 2021-12-15 12:29:57,157 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:29:57,160 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:29:57,160 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:29:59,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:29:59,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:29:59,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:07,171 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:07,171 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.53 2021-12-15 12:30:07,186 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:07,187 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:07,187 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:09,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:09,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:09,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:17,200 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:17,200 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.53 2021-12-15 12:30:17,219 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:17,220 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:17,221 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:19,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:19,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:19,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:27,233 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:27,233 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.53 2021-12-15 12:30:27,253 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:27,256 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:27,257 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:29,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.53 % 2021-12-15 12:30:29,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:29,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:37,267 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.73 % 2021-12-15 12:30:37,269 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.73 2021-12-15 12:30:37,285 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:37,288 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:37,288 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:39,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.73 % 2021-12-15 12:30:39,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:39,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:47,299 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.73 % 2021-12-15 12:30:47,299 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.73 2021-12-15 12:30:47,315 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:47,316 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:47,316 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:49,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.73 % 2021-12-15 12:30:49,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:49,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:57,328 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-15 12:30:57,328 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-15 12:30:57,344 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:30:57,346 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:30:57,346 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:30:59,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-15 12:30:59,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:30:59,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:07,358 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-15 12:31:07,358 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-15 12:31:07,374 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:31:07,377 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:31:07,377 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:09,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-15 12:31:09,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:31:09,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:17,388 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.12 % 2021-12-15 12:31:17,388 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 20.12 2021-12-15 12:31:17,404 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:31:17,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:31:17,407 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:19,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.12 % 2021-12-15 12:31:19,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:31:19,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:27,418 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.12 % 2021-12-15 12:31:27,418 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 20.12 2021-12-15 12:31:27,434 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:31:27,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:31:27,437 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:29,572 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.12% in 2164.59800005 seconds 2021-12-15 12:31:29,582 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 12:31:29,582 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 12:31:29,585 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 12:31:29,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:31:29,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 12:31:29,921 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:31:29,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 12:31:29,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 12:31:29,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 12:31:29,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 12:31:29,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 12:31:29,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 12:31:30,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 12:31:30,023 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 12:31:30,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 12:31:30,079 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['33538 memcached\n'], []) 2021-12-15 12:31:30,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 12:31:30,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 12:31:30,183 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['33538 memcached\n'], []) 2021-12-15 12:31:30,184 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 12:31:30,187 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 12:31:30,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:31:30,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 12:31:30,586 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:31:30,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 12:31:30,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 12:31:30,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 12:31:30,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 12:31:30,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 12:31:30,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 12:31:30,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 12:31:30,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:31:30,713 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 12:31:30,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:31:30,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:30,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:30,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:30,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:30,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:30,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:30,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:30,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:30,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:30,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:31,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:31:31,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:31:31,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:31,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:31,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 12:31:31,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 12:31:31,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:31:31,115 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 12:31:31,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:31:31,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:31,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:31:31,154 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 12:31:31,155 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 12:31:31,160 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 12:31:31,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:31:31,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 12:31:31,601 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:31:31,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 12:31:31,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 12:31:31,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 12:31:31,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 12:31:31,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 12:31:31,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 12:31:31,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 12:31:31,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:31:31,756 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 12:31:31,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:31:31,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:31,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:31,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:31,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:31,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:32,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:32,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:32,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:32,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:32,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:32,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:31:32,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:31:32,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:31:32,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 12:31:32,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:31:32,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:32,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:31:32,331 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 12:31:32,331 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 12:31:32,334 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 12:31:32,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:31:32,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 12:31:32,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:31:32,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 12:31:32,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 12:31:32,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 12:31:32,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 12:31:32,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 12:31:32,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 12:31:32,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 12:31:32,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:31:32,982 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 12:31:33,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:31:33,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:33,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:33,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:33,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:33,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:31:33,118 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 12:31:33,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:31:33,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:33,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:31:33,153 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 12:31:33,154 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 12:31:33,157 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 12:31:33,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:31:33,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 12:31:33,650 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:31:33,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 12:31:33,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 12:31:33,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 12:31:33,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 12:31:33,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 12:31:33,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 12:31:33,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 12:31:33,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:31:33,793 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 12:31:33,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:31:33,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:33,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:33,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:33,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:31:34,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:34,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:31:34,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:34,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:31:34,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:31:34,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 12:31:34,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:31:34,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:31:34,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:31:34,242 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 12:31:34,242 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 12:31:34,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 12:31:34,249 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 12:31:37,450 | test | DEBUG | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.12 % 2021-12-15 12:31:37,450 | test | INFO | pool-3-thread-20 | [task:check:385] Rebalance - status: running, progress: 20.12 2021-12-15 12:31:37,470 | infra | DEBUG | pool-3-thread-20 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:31:37,471 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:31:37,471 | infra | DEBUG | pool-3-thread-20 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:31:47,482 | test | ERROR | pool-3-thread-20 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'00594de6b8d6e3206f5135d331f3097a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c17a489e08b5406a6211dfb6d10cf571', u'status': u'notRunning'} - rebalance failed 2021-12-15 12:31:47,506 | test | INFO | pool-3-thread-20 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 12:31:47,506 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301264L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.264Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nW1215 05:16:06.940414 75546 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 06:37:23.493861 33679 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 08:12:20.866575 33677 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 10:17:14.209003 33682 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 11:19:57.221807 33681 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 12:31:47,506 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301248L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.248Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30088.109>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,506}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {set_vbucket,151,active,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.8660.106>,\n {update_vbucket_state,151,active,paused,\n undefined,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]}},\n infinity]}}}}}.\nRebalance Operation Id = 274721e656eeff3e99e26fb4ac58d12d'} 2021-12-15 12:31:47,507 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301219L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.219Z', u'text': u'Worker <0.24772.109> (for action {move,\n {151,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.30088.109>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 163}]},\n {mc_client_binary,\n stats,4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 455}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 506}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 225}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {set_vbucket,\n 151,\n active,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.8660.106>,\n {update_vbucket_state,\n 151,\n active,\n paused,\n undefined,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]}},\n infinity]}}}}'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301204L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.204Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639598052333L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:54:12.333Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1639598039333L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:53:59.333Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.68\' in 0 seconds.'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639598038995L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:53:58.995Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1639598038918L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:53:58.918Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639598038784L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:53:58.784Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 274721e656eeff3e99e26fb4ac58d12d"} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [rest_client:print_UI_logs:2805] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1639598038748L, u'shortText': u'message', u'serverTime': u'2021-12-15T11:53:58.748Z', u'text': u'Node ns_1@172.23.110.68 joined cluster'} 2021-12-15 12:31:47,509 | test | ERROR | pool-3-thread-20 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'00594de6b8d6e3206f5135d331f3097a', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c17a489e08b5406a6211dfb6d10cf571', u'status': u'notRunning'} - rebalance failed 2021-12-15 12:32:34,263 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 12:32:34,263 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 12:32:34,276 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:34,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:34,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:36,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:36,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:36,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:38,322 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:38,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:38,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:40,348 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:40,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:40,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:42,368 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:42,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:42,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:44,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:44,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:44,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:46,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:46,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:46,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:48,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:48,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:48,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:50,454 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:50,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:50,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:52,471 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:52,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:52,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:54,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:54,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:54,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:56,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:56,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:56,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:32:58,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:32:58,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:32:58,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:00,558 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:00,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:00,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:02,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:02,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:02,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:04,599 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:04,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:04,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:06,619 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:06,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:06,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:08,640 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:08,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:08,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:10,657 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:10,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:10,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:12,677 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:12,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:12,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:14,697 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:14,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:14,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:16,716 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:16,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:16,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:18,737 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:18,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:18,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:20,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:20,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:20,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:22,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:22,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:22,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:24,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:24,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:24,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:26,825 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:26,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:26,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:28,846 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:28,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:28,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:30,867 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:30,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:30,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:32,885 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:32,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:32,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:34,907 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:34,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:34,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:36,927 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:36,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:36,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:38,947 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:38,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:38,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:40,967 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:40,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:40,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:42,992 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:42,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:42,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:45,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:45,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:45,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:47,036 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:47,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:47,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:49,056 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:49,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:49,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:51,076 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:51,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:51,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:53,098 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:53,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:53,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:55,118 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:55,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:55,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:57,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:57,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:57,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:33:59,161 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:33:59,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:33:59,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:01,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:01,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:01,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:03,207 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:03,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:03,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:05,230 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:05,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:05,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:07,250 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:07,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:07,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:09,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:09,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:09,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:11,289 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:11,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:11,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:13,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:13,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:13,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:15,331 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:15,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:15,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:17,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:17,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:17,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:19,369 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:19,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:19,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:21,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:21,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:21,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:23,410 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:23,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:23,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:25,448 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:25,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:25,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:27,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:27,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:27,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:29,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:29,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:29,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:31,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:31,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:31,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:33,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:33,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:33,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:35,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:35,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:35,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:37,598 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:37,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:37,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:39,621 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:39,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:39,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:41,640 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:41,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:41,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:43,665 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:43,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:43,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:45,693 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:45,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:45,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:47,714 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:47,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:47,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:49,733 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:49,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:49,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:51,752 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:51,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:51,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:53,773 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:53,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:53,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:55,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:55,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:55,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:57,825 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:57,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:57,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:34:59,848 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:34:59,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:34:59,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:01,871 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:01,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:01,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:03,891 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:03,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:03,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:05,911 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:05,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:05,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:07,930 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:07,936 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:07,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:09,950 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:09,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:09,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:11,970 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:11,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:11,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:13,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:13,999 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:13,999 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:16,012 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:16,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:16,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:18,032 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:18,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:18,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:20,052 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:20,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:20,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:22,072 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:22,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:22,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:24,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:24,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:24,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:26,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:26,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:26,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:28,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:28,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:28,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:30,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:30,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:30,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:32,168 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:32,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:32,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:34,207 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:34,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:34,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:36,230 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:36,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:36,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:38,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:38,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:38,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:40,278 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:40,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:40,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:42,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:42,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:42,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:44,319 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:44,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:44,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:46,342 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:46,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:46,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:48,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:48,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:48,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:50,387 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:50,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:50,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:52,404 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:52,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:52,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:54,426 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:54,433 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:54,433 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:56,444 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:56,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:56,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:35:58,463 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:35:58,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:35:58,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:00,483 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:00,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:00,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:02,505 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:02,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:02,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:04,525 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:04,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:04,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:06,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:06,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:06,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:08,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:08,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:08,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:10,584 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:10,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:10,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:12,605 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:12,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:12,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:14,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:14,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:14,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:16,650 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:16,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:16,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:18,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:18,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:18,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:20,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:20,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:20,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:22,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:22,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:22,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:24,733 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:24,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:24,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:26,753 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:26,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:26,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:28,770 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:28,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:28,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:30,792 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:30,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:30,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:32,816 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:32,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:32,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:34,842 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:34,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:34,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:36,865 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:36,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:36,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:38,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:38,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:38,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:40,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:40,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:40,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:42,924 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:42,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:42,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:44,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:44,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:44,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:46,963 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:46,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:46,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:48,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:48,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:48,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:51,005 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:51,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:51,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:53,025 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:53,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:53,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:55,046 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:55,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:55,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:57,063 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:57,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:57,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:36:59,095 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:36:59,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:36:59,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:01,115 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:01,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:01,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:03,134 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:03,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:03,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:05,155 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:05,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:05,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:07,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:07,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:07,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:09,196 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:09,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:09,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:11,217 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:11,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:11,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:13,240 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:13,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:13,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:15,260 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:15,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:15,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:17,280 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:17,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:17,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:19,305 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:19,313 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:19,313 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:21,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:21,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:21,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:23,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:23,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:23,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:25,374 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:25,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:25,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:27,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:27,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:27,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:29,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:29,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:29,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:31,438 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:31,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:31,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:33,461 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:33,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:33,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:35,483 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:35,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:35,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:37,507 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:37,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:37,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:39,529 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:39,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:39,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:41,549 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 12:37:41,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 12:37:41,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 12:37:43,569 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 12:37:43,578 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 12:37:43,578 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 12:37:43,588 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 12:37:43,604 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-15 12:37:43,604 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 12:37:43,617 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-15 12:37:43,627 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 12:37:43,627 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 12:37:43,637 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 12:37:43,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 12:37:43,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:37:43,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 12:37:43,977 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:37:43,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 12:37:43,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 12:37:44,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 12:37:44,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 12:37:44,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 12:37:44,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 12:37:44,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 12:37:44,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:37:44,101 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 12:37:44,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:37:44,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:44,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:44,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:44,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:44,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:44,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:44,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:44,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:44,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:44,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:44,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:44,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:44,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:44,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:44,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 12:37:44,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 12:37:44,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:37:44,573 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 12:37:44,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:37:44,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:44,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:37:44,627 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 12:37:44,627 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 12:37:44,631 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 12:37:44,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:37:44,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 12:37:45,017 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:37:45,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 12:37:45,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 12:37:45,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 12:37:45,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 12:37:45,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 12:37:45,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 12:37:45,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 12:37:45,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:37:45,128 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 12:37:45,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:37:45,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:45,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:45,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:45,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:45,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:45,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:45,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:45,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:45,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:45,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:45,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:45,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:45,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:37:45,423 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 12:37:45,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:37:45,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:45,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:37:45,457 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 12:37:45,457 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 12:37:45,460 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 12:37:45,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:37:45,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 12:37:45,890 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:37:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 12:37:45,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 12:37:45,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 12:37:45,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 12:37:45,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 12:37:45,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 12:37:45,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 12:37:45,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:37:46,000 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 12:37:46,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:37:46,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:46,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:46,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:46,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:46,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:37:46,128 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 12:37:46,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:37:46,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:46,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:37:46,167 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 12:37:46,167 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 12:37:46,170 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 12:37:46,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 12:37:46,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 12:37:46,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 12:37:46,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 12:37:46,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 12:37:46,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 12:37:46,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 12:37:46,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 12:37:46,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 12:37:46,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 12:37:46,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 12:37:46,732 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 12:37:46,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 12:37:46,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:46,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:46,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:46,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 12:37:46,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:46,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:46,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:46,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 12:37:46,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:46,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 12:37:47,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:47,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 12:37:47,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 12:37:47,036 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 12:37:47,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 12:37:47,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 12:37:47,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 12:37:47,073 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 12:37:47,073 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 12:37:47,075 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.68]_OUT=[]_1639598034.91 2021-12-15 12:37:47,075 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-15 12:37:47,105 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1639600667.08 2021-12-15 12:37:47,105 | infra | INFO | pool-3-thread-21 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[]_1639600667.08' started 2021-12-15 12:37:47,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 12:37:47,118 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 12:37:47,134 | test | DEBUG | pool-3-thread-21 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-15 12:37:47,188 | test | DEBUG | pool-3-thread-21 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 12:37:47,191 | test | INFO | pool-3-thread-21 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 6.53298475915 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.175080245112 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 0.997970700247 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 3.70215727209 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-15 12:37:47,200 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:37:47,200 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:37:47,217 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:37:47,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:37:47,220 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:37:57,230 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:37:57,230 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:37:57,249 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:37:57,252 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:37:57,252 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:07,262 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:38:07,262 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:38:07,276 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:07,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:07,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:17,289 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:38:17,289 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:38:17,303 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:17,306 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:17,306 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:27,315 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:38:27,316 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:38:27,331 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:27,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:27,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:37,342 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:38:37,342 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:38:37,358 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:37,359 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:37,359 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:47,369 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 12:38:47,369 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 12:38:47,384 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:47,387 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:47,387 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:38:57,398 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.49 % 2021-12-15 12:38:57,398 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.49 2021-12-15 12:38:57,418 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:38:57,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:38:57,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:07,431 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.49 % 2021-12-15 12:39:07,434 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.49 2021-12-15 12:39:07,447 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:07,450 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:07,450 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:17,463 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.49 % 2021-12-15 12:39:17,463 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.49 2021-12-15 12:39:17,483 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:17,484 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:17,484 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:27,499 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.49 % 2021-12-15 12:39:27,499 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.49 2021-12-15 12:39:27,519 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:27,522 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:27,522 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:37,533 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.49 % 2021-12-15 12:39:37,535 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.49 2021-12-15 12:39:37,555 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:37,556 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:37,556 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:47,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:39:47,128 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.99 2021-12-15 12:39:47,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:39:47,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:39:47,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:47,568 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:39:47,568 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.99 2021-12-15 12:39:47,582 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:47,584 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:47,584 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:57,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:39:57,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:39:57,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:39:57,595 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:39:57,595 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.99 2021-12-15 12:39:57,615 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:39:57,617 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:39:57,617 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:07,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:40:07,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:07,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:07,628 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.99 % 2021-12-15 12:40:07,628 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 0.99 2021-12-15 12:40:07,645 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:07,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:07,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:17,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.23 % 2021-12-15 12:40:17,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:17,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:17,661 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.23 % 2021-12-15 12:40:17,661 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.23 2021-12-15 12:40:17,680 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:17,681 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:17,681 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:27,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.47 % 2021-12-15 12:40:27,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:27,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:27,691 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.47 % 2021-12-15 12:40:27,691 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.47 2021-12-15 12:40:27,717 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:27,720 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:27,720 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:37,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.47 % 2021-12-15 12:40:37,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:37,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:37,730 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.47 % 2021-12-15 12:40:37,730 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.47 2021-12-15 12:40:37,750 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:37,752 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:37,752 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:47,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:40:47,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:47,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:47,763 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:40:47,763 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-15 12:40:47,785 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:47,786 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:47,788 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:57,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:40:57,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:40:57,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:40:57,799 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:40:57,799 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-15 12:40:57,819 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:40:57,821 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:40:57,821 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:07,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:41:07,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:07,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:07,832 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:41:07,832 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-15 12:41:07,852 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:07,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:07,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:17,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-15 12:41:17,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:17,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:17,881 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.97 % 2021-12-15 12:41:17,881 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.97 2021-12-15 12:41:17,901 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:17,904 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:17,904 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:27,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.97 % 2021-12-15 12:41:27,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:27,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:27,917 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.97 % 2021-12-15 12:41:27,917 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 1.97 2021-12-15 12:41:27,934 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:27,937 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:27,937 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:37,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.22 % 2021-12-15 12:41:37,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:37,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:37,950 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.22 % 2021-12-15 12:41:37,950 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.22 2021-12-15 12:41:37,966 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:37,967 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:37,969 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:47,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.22 % 2021-12-15 12:41:47,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:47,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:47,980 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.22 % 2021-12-15 12:41:47,980 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.22 2021-12-15 12:41:47,999 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:48,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:48,003 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:57,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 12:41:57,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:41:57,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:41:58,015 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 12:41:58,016 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.45 2021-12-15 12:41:58,030 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:41:58,035 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:41:58,035 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:07,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 12:42:07,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:07,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:08,046 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 12:42:08,046 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.45 2021-12-15 12:42:08,062 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:08,066 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:08,066 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:17,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:17,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:17,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:18,078 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:18,078 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.7 2021-12-15 12:42:18,095 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:18,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:18,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:27,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:27,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:27,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:28,108 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:28,109 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.7 2021-12-15 12:42:28,124 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:28,127 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:28,127 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:37,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:37,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:37,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:38,137 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.70 % 2021-12-15 12:42:38,138 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 2.7 2021-12-15 12:42:38,151 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:38,154 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:38,154 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:47,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:42:47,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:47,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:48,164 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:42:48,165 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 12:42:48,180 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:48,184 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:48,184 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:57,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:42:57,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:42:57,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:42:58,196 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:42:58,196 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 12:42:58,210 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:42:58,213 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:42:58,213 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:07,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:43:07,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:07,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:08,223 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:43:08,224 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 12:43:08,240 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:08,243 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:08,243 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:17,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 12:43:17,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:17,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:18,253 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.45 % 2021-12-15 12:43:18,253 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.45 2021-12-15 12:43:18,276 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:18,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:18,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:27,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:27,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:27,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:28,290 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:28,292 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:43:28,309 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:28,312 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:28,312 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:37,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:37,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:37,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:38,323 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:38,325 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:43:38,338 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:38,341 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:38,341 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:47,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:47,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:47,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:48,351 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:48,351 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:43:48,364 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:48,365 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:48,365 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:57,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:57,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:43:57,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:43:58,377 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:43:58,378 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:43:58,391 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:43:58,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:43:58,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:07,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:44:07,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:07,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:08,401 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:44:08,403 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:44:08,417 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:08,418 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:08,418 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:17,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:44:17,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:17,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:18,427 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.68 % 2021-12-15 12:44:18,428 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.68 2021-12-15 12:44:18,441 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:18,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:18,444 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:27,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.94 % 2021-12-15 12:44:27,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:27,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:28,454 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.94 % 2021-12-15 12:44:28,454 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 3.94 2021-12-15 12:44:28,470 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:28,470 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:28,471 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:37,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.19 % 2021-12-15 12:44:37,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:37,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:38,483 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.43 % 2021-12-15 12:44:38,483 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.43 2021-12-15 12:44:38,497 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:38,500 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:38,500 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:47,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.43 % 2021-12-15 12:44:47,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:47,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:48,510 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.43 % 2021-12-15 12:44:48,512 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.43 2021-12-15 12:44:48,526 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:48,529 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:48,529 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:57,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:44:57,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:44:57,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:44:58,542 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:44:58,542 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.67 2021-12-15 12:44:58,558 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:44:58,559 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:44:58,559 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:07,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:07,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:07,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:08,572 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:08,572 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.67 2021-12-15 12:45:08,591 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:08,594 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:08,594 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:17,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:17,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:17,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:18,605 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:18,605 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.67 2021-12-15 12:45:18,619 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:18,622 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:18,622 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:27,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:27,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:27,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:28,632 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:28,634 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.67 2021-12-15 12:45:28,648 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:28,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:28,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:37,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:37,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:37,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:38,661 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.67 % 2021-12-15 12:45:38,661 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.67 2021-12-15 12:45:38,676 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:38,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:38,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:47,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.92 % 2021-12-15 12:45:47,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:47,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:48,690 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.92 % 2021-12-15 12:45:48,690 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.92 2021-12-15 12:45:48,717 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:48,720 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:48,720 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:57,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.92 % 2021-12-15 12:45:57,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:45:57,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:45:58,732 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.92 % 2021-12-15 12:45:58,732 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 4.92 2021-12-15 12:45:58,752 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:45:58,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:45:58,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:07,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.17 % 2021-12-15 12:46:07,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:07,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:08,776 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.17 % 2021-12-15 12:46:08,776 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.17 2021-12-15 12:46:08,792 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:08,793 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:08,793 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:17,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.42 % 2021-12-15 12:46:17,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:17,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:18,806 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.42 % 2021-12-15 12:46:18,806 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.42 2021-12-15 12:46:18,828 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:18,832 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:18,832 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:27,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:27,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:27,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:28,845 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:28,845 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.66 2021-12-15 12:46:28,865 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:28,868 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:28,868 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:37,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:37,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:37,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:38,880 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:38,880 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.66 2021-12-15 12:46:38,895 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:38,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:38,898 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:47,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:47,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:47,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:48,908 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:48,908 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.66 2021-12-15 12:46:48,926 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:48,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:48,928 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:57,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:57,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:46:57,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:46:58,940 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:46:58,940 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.66 2021-12-15 12:46:58,956 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:46:58,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:46:58,959 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:07,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:47:07,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:07,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:08,969 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-15 12:47:08,969 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.66 2021-12-15 12:47:08,983 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:08,986 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:08,986 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:17,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.90 % 2021-12-15 12:47:17,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:17,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:19,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.90 % 2021-12-15 12:47:19,000 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 5.9 2021-12-15 12:47:19,016 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:19,017 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:19,017 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:27,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.14 % 2021-12-15 12:47:27,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:27,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:29,029 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.14 % 2021-12-15 12:47:29,029 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.14 2021-12-15 12:47:29,043 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:29,048 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:29,048 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:37,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-15 12:47:38,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:38,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:39,058 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-15 12:47:39,059 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.39 2021-12-15 12:47:39,073 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:39,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:39,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:48,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:47:48,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:48,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:49,089 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:47:49,089 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.63 2021-12-15 12:47:49,108 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:49,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:49,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:58,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:47:58,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:47:58,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:47:59,122 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:47:59,122 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.63 2021-12-15 12:47:59,140 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:47:59,142 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:47:59,142 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:08,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:48:08,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:08,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:09,154 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:48:09,154 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.63 2021-12-15 12:48:09,168 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:09,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:09,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:18,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:48:18,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:18,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:19,183 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-15 12:48:19,184 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.63 2021-12-15 12:48:19,197 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:19,200 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:19,200 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:28,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:28,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:28,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:29,213 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:29,213 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.89 2021-12-15 12:48:29,227 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:29,232 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:29,232 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:38,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:38,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:38,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:39,243 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:39,243 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.89 2021-12-15 12:48:39,259 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:39,263 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:39,263 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:48,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:48,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:48,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:49,273 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:49,273 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.89 2021-12-15 12:48:49,289 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:49,292 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:49,292 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:58,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:58,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:48:58,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:48:59,303 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.89 % 2021-12-15 12:48:59,303 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 6.89 2021-12-15 12:48:59,322 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:48:59,325 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:48:59,325 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:08,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.37 % 2021-12-15 12:49:08,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:08,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:09,336 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.37 % 2021-12-15 12:49:09,336 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.37 2021-12-15 12:49:09,355 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:09,358 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:09,358 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:18,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.37 % 2021-12-15 12:49:18,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:18,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:19,368 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:49:19,368 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:49:19,384 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:19,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:19,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:28,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:49:28,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:28,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:29,397 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 12:49:29,397 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 12:49:29,411 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:29,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:29,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:38,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:38,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:38,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:39,426 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:39,426 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.86 2021-12-15 12:49:39,440 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:39,441 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:39,441 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:48,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:48,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:48,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:49,453 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:49,453 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.86 2021-12-15 12:49:49,467 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:49,469 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:49,469 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:58,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:58,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:49:58,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:49:59,477 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:49:59,479 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.86 2021-12-15 12:49:59,492 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:49:59,493 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:49:59,493 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:08,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:50:08,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:08,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:09,506 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.86 % 2021-12-15 12:50:09,506 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 7.86 2021-12-15 12:50:09,526 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:09,529 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:09,529 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:18,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.12 % 2021-12-15 12:50:18,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:18,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:19,539 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.12 % 2021-12-15 12:50:19,539 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.12 2021-12-15 12:50:19,552 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:19,555 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:19,555 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:28,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:28,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:28,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:29,566 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:29,566 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.35 2021-12-15 12:50:29,582 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:29,584 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:29,584 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:38,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:38,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:38,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:39,595 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:39,595 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.35 2021-12-15 12:50:39,615 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:39,618 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:39,618 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:48,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:48,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:48,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:49,630 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.35 % 2021-12-15 12:50:49,630 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.35 2021-12-15 12:50:49,644 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:49,647 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:49,647 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:58,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 12:50:58,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:50:58,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:50:59,657 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 12:50:59,657 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 12:50:59,676 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:50:59,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:50:59,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:08,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:08,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:08,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:09,690 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:09,690 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-15 12:51:09,710 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:09,713 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:09,713 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:18,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:18,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:18,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:19,723 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:19,726 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-15 12:51:19,740 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:19,743 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:19,743 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:28,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:28,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:28,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:29,755 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:29,755 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-15 12:51:29,775 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:29,776 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:29,776 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:38,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:38,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:38,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:39,790 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-15 12:51:39,792 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-15 12:51:39,811 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:39,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:39,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:48,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.33 % 2021-12-15 12:51:48,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:48,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:49,825 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.33 % 2021-12-15 12:51:49,826 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.33 2021-12-15 12:51:49,846 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:49,849 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:49,849 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:58,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.33 % 2021-12-15 12:51:58,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:51:58,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:51:59,862 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.33 % 2021-12-15 12:51:59,862 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.33 2021-12-15 12:51:59,884 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:51:59,888 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:51:59,888 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:08,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:08,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:08,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:09,903 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:09,904 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 12:52:09,924 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:52:09,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:52:09,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:18,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:18,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:18,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:19,940 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:19,940 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 12:52:19,957 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:52:19,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:52:19,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:28,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:28,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:28,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:29,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 12:52:29,970 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 12:52:29,984 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:52:29,986 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:52:29,986 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:38,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-15 12:52:38,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:38,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:39,999 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-15 12:52:39,999 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-15 12:52:40,015 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:52:40,019 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:52:40,019 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:48,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-15 12:52:48,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:48,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:50,032 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-15 12:52:50,032 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-15 12:52:50,052 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:52:50,053 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:52:50,053 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:52:58,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:52:58,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:52:58,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:00,063 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:53:00,065 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.08 2021-12-15 12:53:00,079 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:00,084 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:00,084 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:08,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:53:08,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:08,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:10,094 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:53:10,095 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.08 2021-12-15 12:53:10,109 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:10,112 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:10,112 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:18,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:53:18,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:18,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:20,122 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 12:53:20,122 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.08 2021-12-15 12:53:20,138 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:20,141 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:20,141 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:28,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-15 12:53:28,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:28,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:30,153 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-15 12:53:30,154 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.32 2021-12-15 12:53:30,174 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:30,176 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:30,177 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:38,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-15 12:53:38,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:38,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:40,187 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-15 12:53:40,187 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.32 2021-12-15 12:53:40,201 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:40,204 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:40,204 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:48,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-15 12:53:48,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:48,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:50,214 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-15 12:53:50,214 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.56 2021-12-15 12:53:50,230 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:53:50,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:53:50,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:53:58,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-15 12:53:58,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:53:58,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:00,247 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-15 12:54:00,249 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 10.56 2021-12-15 12:54:00,269 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:00,273 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:00,273 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:08,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:08,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:08,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:10,283 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:10,285 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.06 2021-12-15 12:54:10,302 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:10,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:10,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:18,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:18,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:18,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:20,315 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:20,315 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.06 2021-12-15 12:54:20,331 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:20,334 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:20,334 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:28,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:28,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:28,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:30,355 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:30,357 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.06 2021-12-15 12:54:30,371 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:30,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:30,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:38,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:38,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:38,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:40,385 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:40,385 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.06 2021-12-15 12:54:40,403 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:40,405 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:40,405 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:48,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:48,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:48,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:50,417 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 12:54:50,417 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.06 2021-12-15 12:54:50,434 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:54:50,437 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:54:50,437 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:54:58,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.29 % 2021-12-15 12:54:58,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:54:58,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:00,450 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.29 % 2021-12-15 12:55:00,450 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.29 2021-12-15 12:55:00,463 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:00,466 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:00,467 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:08,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.29 % 2021-12-15 12:55:08,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:08,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:10,477 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.29 % 2021-12-15 12:55:10,477 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.29 2021-12-15 12:55:10,493 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:10,496 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:10,496 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:18,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.54 % 2021-12-15 12:55:18,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:18,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:20,509 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.79 % 2021-12-15 12:55:20,509 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.79 2021-12-15 12:55:20,525 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:20,528 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:20,529 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:28,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.79 % 2021-12-15 12:55:28,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:28,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:30,538 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.79 % 2021-12-15 12:55:30,539 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 11.79 2021-12-15 12:55:30,552 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:30,555 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:30,555 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:38,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:55:38,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:38,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:40,569 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:55:40,569 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.04 2021-12-15 12:55:40,584 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:40,585 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:40,585 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:48,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:55:48,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:48,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:50,595 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:55:50,595 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.04 2021-12-15 12:55:50,611 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:55:50,612 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:55:50,612 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:55:58,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:55:58,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:55:58,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:00,625 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:56:00,625 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.04 2021-12-15 12:56:00,638 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:00,641 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:00,641 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:08,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:56:08,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:08,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:10,651 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.04 % 2021-12-15 12:56:10,653 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.04 2021-12-15 12:56:10,667 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:10,670 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:10,670 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:19,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:19,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:19,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:20,680 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:20,680 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.28 2021-12-15 12:56:20,696 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:20,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:20,698 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:29,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:29,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:29,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:30,710 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:30,710 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.28 2021-12-15 12:56:30,729 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:30,732 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:30,732 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:39,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:39,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:39,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:40,743 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:40,743 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.28 2021-12-15 12:56:40,763 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:40,765 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:40,765 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:49,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 12:56:49,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:49,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:50,776 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.52 % 2021-12-15 12:56:50,776 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 12.52 2021-12-15 12:56:50,792 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:56:50,793 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:56:50,795 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:56:59,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.78 % 2021-12-15 12:56:59,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:56:59,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:00,805 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:00,805 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:00,825 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:00,826 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:00,826 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:09,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:09,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:09,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:10,838 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:10,838 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:10,858 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:10,861 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:10,861 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:19,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:19,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:19,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:20,872 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:20,872 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:20,891 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:20,894 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:20,894 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:29,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:29,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:29,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:30,905 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:30,905 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:30,924 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:30,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:30,927 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:39,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:39,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:39,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:40,940 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:40,940 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:40,957 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:40,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:40,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:49,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:49,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:49,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:50,970 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-15 12:57:50,970 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-15 12:57:50,984 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:57:50,986 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:57:50,987 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:57:59,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.26 % 2021-12-15 12:57:59,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:57:59,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:01,013 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.26 % 2021-12-15 12:58:01,013 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.26 2021-12-15 12:58:01,035 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:01,038 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:01,038 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:09,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:09,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:09,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:11,049 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:11,049 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.51 2021-12-15 12:58:11,066 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:11,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:11,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:19,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:19,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:19,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:21,079 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:21,079 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.51 2021-12-15 12:58:21,098 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:21,102 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:21,102 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:29,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:29,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:29,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:31,115 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:31,115 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 13.51 2021-12-15 12:58:31,132 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:31,137 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:31,137 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:39,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.51 % 2021-12-15 12:58:39,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:39,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:41,148 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:58:41,148 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.01 2021-12-15 12:58:41,164 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:41,165 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:41,167 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:49,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:58:49,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:49,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:51,177 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:58:51,177 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.01 2021-12-15 12:58:51,193 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:58:51,196 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:58:51,197 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:58:59,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:58:59,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:58:59,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:01,207 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:59:01,207 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.01 2021-12-15 12:59:01,223 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:01,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:01,226 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:09,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:59:09,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:09,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:11,236 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.01 % 2021-12-15 12:59:11,236 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.01 2021-12-15 12:59:11,250 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:11,253 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:11,253 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:19,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.25 % 2021-12-15 12:59:19,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:19,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:21,265 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.25 % 2021-12-15 12:59:21,265 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.25 2021-12-15 12:59:21,280 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:21,283 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:21,283 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:29,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:29,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:29,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:31,296 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:31,298 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.49 2021-12-15 12:59:31,315 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:31,319 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:31,319 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:39,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:39,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:39,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:41,331 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:41,332 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.49 2021-12-15 12:59:41,348 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:41,351 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:41,351 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:49,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:49,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:49,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:51,365 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.49 % 2021-12-15 12:59:51,365 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.49 2021-12-15 12:59:51,382 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 12:59:51,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 12:59:51,385 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 12:59:59,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.74 % 2021-12-15 12:59:59,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 12:59:59,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:01,397 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.74 % 2021-12-15 13:00:01,398 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.74 2021-12-15 13:00:01,411 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:01,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:01,414 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:09,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.74 % 2021-12-15 13:00:09,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:09,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:11,426 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.74 % 2021-12-15 13:00:11,426 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.74 2021-12-15 13:00:11,446 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:11,448 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:11,460 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:19,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:19,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:19,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:21,470 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:21,470 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.99 2021-12-15 13:00:21,486 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:21,489 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:21,489 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:29,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:29,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:29,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:31,499 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:31,499 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.99 2021-12-15 13:00:31,515 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:31,516 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:31,517 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:39,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:39,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:39,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:41,529 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 13:00:41,530 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 14.99 2021-12-15 13:00:41,546 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:41,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:41,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:49,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:00:49,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:49,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:51,561 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:00:51,561 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.48 2021-12-15 13:00:51,576 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:00:51,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:00:51,578 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:00:59,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:00:59,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:00:59,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:01,589 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:01,589 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.48 2021-12-15 13:01:01,605 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:01,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:01,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:09,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:09,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:09,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:11,618 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:11,618 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.48 2021-12-15 13:01:11,635 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:11,638 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:11,638 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:19,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:19,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:19,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:21,651 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:21,651 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.48 2021-12-15 13:01:21,664 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:21,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:21,667 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:29,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.48 % 2021-12-15 13:01:29,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:29,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:31,677 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.72 % 2021-12-15 13:01:31,677 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.72 2021-12-15 13:01:31,690 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:31,691 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:31,691 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:39,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.72 % 2021-12-15 13:01:39,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:39,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:41,703 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.72 % 2021-12-15 13:01:41,703 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.72 2021-12-15 13:01:41,723 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:41,724 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:41,724 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:49,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.97 % 2021-12-15 13:01:49,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:49,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:51,736 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.97 % 2021-12-15 13:01:51,736 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 15.97 2021-12-15 13:01:51,752 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:01:51,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:01:51,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:01:59,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:01:59,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:01:59,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:01,766 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:01,767 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.22 2021-12-15 13:02:01,786 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:01,788 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:01,788 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:09,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:09,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:09,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:11,801 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:11,801 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.22 2021-12-15 13:02:11,818 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:11,819 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:11,819 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:19,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:19,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:19,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:21,831 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:21,831 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.22 2021-12-15 13:02:21,845 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:21,846 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:21,846 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:29,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.22 % 2021-12-15 13:02:29,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:29,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:31,858 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.46 % 2021-12-15 13:02:31,858 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.46 2021-12-15 13:02:31,877 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:31,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:31,878 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:39,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:02:39,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:39,747 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:41,891 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:02:41,891 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 13:02:41,913 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:41,915 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:41,915 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:49,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:02:49,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:49,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:51,927 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:02:51,927 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 13:02:51,947 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:02:51,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:02:51,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:02:59,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:02:59,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:02:59,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:01,960 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:03:01,960 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 13:03:01,977 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:01,980 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:01,980 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:09,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:03:09,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:09,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:11,992 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:03:11,992 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 13:03:12,007 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:12,009 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:12,009 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:19,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:03:19,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:19,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:22,019 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 13:03:22,019 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 13:03:22,033 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:22,035 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:22,036 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:29,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.95 % 2021-12-15 13:03:29,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:29,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:32,045 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.95 % 2021-12-15 13:03:32,046 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 16.95 2021-12-15 13:03:32,062 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:32,065 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:32,065 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:39,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 13:03:39,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:39,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:42,075 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 13:03:42,075 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 13:03:42,091 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:42,094 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:42,094 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:49,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 13:03:49,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:49,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:52,102 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 13:03:52,104 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 13:03:52,118 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:03:52,121 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:03:52,121 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:03:59,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:03:59,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:03:59,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:02,131 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:02,131 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 13:04:02,145 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:02,148 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:02,148 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:09,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:09,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:04:09,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:12,160 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:12,160 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 13:04:12,176 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:12,177 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:12,178 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:19,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:19,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:04:19,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:22,188 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:22,190 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 13:04:22,206 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:22,207 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:22,209 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:29,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:29,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:04:29,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:32,220 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:32,220 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 13:04:32,236 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:32,237 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:32,237 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:39,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:39,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:04:39,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:42,250 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 13:04:42,250 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 13:04:42,266 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:42,267 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:42,267 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:49,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 13:04:50,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:04:50,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:04:52,278 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 13:04:52,278 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.94 2021-12-15 13:04:52,292 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:04:52,295 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:04:52,295 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:00,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 13:05:00,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:00,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:02,306 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 13:05:02,308 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 17.94 2021-12-15 13:05:02,325 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:02,328 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:02,328 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:10,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 13:05:10,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:10,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:12,345 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 13:05:12,345 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 13:05:12,361 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:12,362 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:12,362 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:20,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.42 % 2021-12-15 13:05:20,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:20,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:22,372 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.42 % 2021-12-15 13:05:22,374 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.42 2021-12-15 13:05:22,387 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:22,391 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:22,391 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:30,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.42 % 2021-12-15 13:05:30,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:30,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:32,401 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.42 % 2021-12-15 13:05:32,401 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.42 2021-12-15 13:05:32,417 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:32,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:32,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:40,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:05:40,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:40,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:42,433 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:05:42,433 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.67 2021-12-15 13:05:42,450 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:42,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:42,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:50,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:05:50,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:05:50,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:05:52,467 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:05:52,467 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.67 2021-12-15 13:05:52,482 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:05:52,484 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:05:52,484 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:00,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:06:00,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:00,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:02,506 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:06:02,506 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.67 2021-12-15 13:06:02,522 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:02,525 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:02,525 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:10,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.67 % 2021-12-15 13:06:10,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:10,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:12,533 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.92 % 2021-12-15 13:06:12,535 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.92 2021-12-15 13:06:12,549 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:12,551 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:12,552 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:20,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.92 % 2021-12-15 13:06:20,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:20,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:22,562 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.92 % 2021-12-15 13:06:22,562 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.92 2021-12-15 13:06:22,578 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:22,581 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:22,581 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:30,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.92 % 2021-12-15 13:06:30,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:30,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:32,589 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.92 % 2021-12-15 13:06:32,591 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 18.92 2021-12-15 13:06:32,605 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:32,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:32,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:40,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.17 % 2021-12-15 13:06:40,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:40,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:42,618 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.17 % 2021-12-15 13:06:42,618 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.17 2021-12-15 13:06:42,632 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:42,634 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:42,635 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:50,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.17 % 2021-12-15 13:06:50,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:06:50,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:06:52,645 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-15 13:06:52,645 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.4 2021-12-15 13:06:52,661 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:06:52,664 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:06:52,664 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:00,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-15 13:07:00,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:00,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:02,673 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-15 13:07:02,674 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.4 2021-12-15 13:07:02,687 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:02,688 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:02,690 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:10,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-15 13:07:10,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:10,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:12,701 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-15 13:07:12,703 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.65 2021-12-15 13:07:12,720 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:12,723 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:12,723 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:20,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-15 13:07:20,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:20,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:22,733 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-15 13:07:22,733 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.65 2021-12-15 13:07:22,747 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:22,750 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:22,750 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:30,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:30,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:30,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:32,760 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:32,762 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.9 2021-12-15 13:07:32,779 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:32,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:32,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:40,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:40,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:40,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:42,792 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:42,792 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.9 2021-12-15 13:07:42,809 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:42,811 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:42,811 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:50,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:50,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:07:50,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:07:52,821 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:07:52,821 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 19.9 2021-12-15 13:07:52,836 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:07:52,838 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:07:52,838 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:00,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-15 13:08:00,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:00,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:02,848 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-15 13:08:02,848 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.15 2021-12-15 13:08:02,865 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:02,867 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:02,867 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:10,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-15 13:08:10,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:10,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:12,877 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-15 13:08:12,877 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.15 2021-12-15 13:08:12,890 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:12,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:12,891 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:20,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:20,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:20,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:22,904 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:22,904 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.39 2021-12-15 13:08:22,924 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:22,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:22,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:30,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:30,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:30,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:32,936 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:32,936 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.39 2021-12-15 13:08:32,957 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:32,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:32,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:40,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:40,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:40,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:42,973 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:42,973 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.39 2021-12-15 13:08:42,990 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:42,993 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:42,993 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:50,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.39 % 2021-12-15 13:08:50,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:08:50,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:08:53,006 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.63 % 2021-12-15 13:08:53,006 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.63 2021-12-15 13:08:53,025 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:08:53,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:08:53,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:00,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.63 % 2021-12-15 13:09:00,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:00,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:03,039 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-15 13:09:03,039 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.88 2021-12-15 13:09:03,058 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:03,059 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:03,059 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:10,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-15 13:09:10,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:10,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:13,072 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-15 13:09:13,072 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 20.88 2021-12-15 13:09:13,088 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:13,092 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:13,092 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:20,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:20,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:20,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:23,104 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:23,104 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.13 2021-12-15 13:09:23,121 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:23,124 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:23,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:30,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:30,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:30,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:33,134 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:33,135 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.13 2021-12-15 13:09:33,154 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:33,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:33,157 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:40,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:40,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:40,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:43,170 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:43,170 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.13 2021-12-15 13:09:43,187 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:43,190 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:43,190 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:50,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 13:09:50,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:09:50,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:09:53,200 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:09:53,200 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.37 2021-12-15 13:09:53,217 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:09:53,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:09:53,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:00,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:00,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:00,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:03,230 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:03,230 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.37 2021-12-15 13:10:03,246 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:03,250 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:03,250 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:10,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:10,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:10,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:13,259 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:13,259 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.37 2021-12-15 13:10:13,275 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:13,278 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:13,278 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:20,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:20,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:20,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:23,289 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:23,289 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.37 2021-12-15 13:10:23,305 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:23,308 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:23,308 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:30,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:30,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:30,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:33,319 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.37 % 2021-12-15 13:10:33,321 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.37 2021-12-15 13:10:33,336 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:33,339 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:33,339 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:40,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.62 % 2021-12-15 13:10:40,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:40,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:43,349 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.62 % 2021-12-15 13:10:43,349 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 21.62 2021-12-15 13:10:43,364 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:43,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:43,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:50,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.11 % 2021-12-15 13:10:50,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:10:50,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:10:53,378 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.11 % 2021-12-15 13:10:53,378 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.11 2021-12-15 13:10:53,392 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:10:53,394 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:10:53,394 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:00,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.11 % 2021-12-15 13:11:00,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:00,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:03,405 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.11 % 2021-12-15 13:11:03,405 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.11 2021-12-15 13:11:03,421 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:03,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:03,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:10,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:10,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:10,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:13,434 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:13,436 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:11:13,451 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:13,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:13,456 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:20,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:20,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:20,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:23,466 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:23,467 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:11:23,480 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:23,482 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:23,483 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:30,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:30,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:30,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:33,492 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:33,492 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:11:33,506 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:33,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:33,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:40,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:40,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:40,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:43,519 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:43,519 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:11:43,535 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:43,538 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:43,538 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:50,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:50,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:11:50,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:11:53,549 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:11:53,549 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:11:53,565 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:11:53,568 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:11:53,568 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:00,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:12:00,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:00,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:03,578 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:12:03,578 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.35 2021-12-15 13:12:03,592 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:03,595 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:03,595 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:10,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.35 % 2021-12-15 13:12:10,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:10,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:13,605 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.60 % 2021-12-15 13:12:13,605 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.6 2021-12-15 13:12:13,621 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:13,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:13,625 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:20,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.85 % 2021-12-15 13:12:20,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:20,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:23,634 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.85 % 2021-12-15 13:12:23,634 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 22.85 2021-12-15 13:12:23,648 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:23,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:23,651 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:30,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 13:12:30,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:30,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:33,661 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 13:12:33,661 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.09 2021-12-15 13:12:33,676 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:33,677 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:33,678 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:40,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 13:12:40,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:40,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:43,688 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 13:12:43,688 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.09 2021-12-15 13:12:43,703 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:43,707 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:43,707 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:50,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 13:12:50,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:12:50,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:12:53,716 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:12:53,717 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.33 2021-12-15 13:12:53,730 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:12:53,732 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:12:53,732 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:00,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:00,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:00,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:03,742 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:03,742 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.33 2021-12-15 13:13:03,756 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:03,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:03,759 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:10,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:10,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:10,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:13,772 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:13,772 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.33 2021-12-15 13:13:13,786 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:13,788 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:13,788 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:21,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:21,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:21,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:23,796 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:23,798 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.33 2021-12-15 13:13:23,815 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:23,818 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:23,818 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:31,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:31,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:31,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:33,829 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.33 % 2021-12-15 13:13:33,829 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.33 2021-12-15 13:13:33,848 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:33,849 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:33,849 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:41,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.58 % 2021-12-15 13:13:41,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:41,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:43,862 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.58 % 2021-12-15 13:13:43,864 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.58 2021-12-15 13:13:43,884 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:43,885 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:43,885 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:51,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.58 % 2021-12-15 13:13:51,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:13:51,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:13:53,898 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.58 % 2021-12-15 13:13:53,898 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 23.58 2021-12-15 13:13:53,917 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:13:53,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:13:53,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:01,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.07 % 2021-12-15 13:14:01,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:01,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:03,930 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.07 % 2021-12-15 13:14:03,930 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.07 2021-12-15 13:14:03,944 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:03,946 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:03,947 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:11,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:11,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:11,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:13,957 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:13,957 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:14:13,976 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:13,977 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:13,979 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:21,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:21,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:21,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:23,990 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:23,990 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:14:24,009 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:24,012 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:24,012 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:31,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:31,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:31,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:34,023 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:34,023 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:14:34,042 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:34,045 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:34,045 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:41,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:41,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:41,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:44,055 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:44,055 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:14:44,069 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:44,072 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:44,072 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:51,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:51,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:14:51,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:14:54,082 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:14:54,082 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:14:54,096 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:14:54,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:14:54,098 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:01,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:15:01,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:01,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:04,111 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:15:04,111 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:15:04,130 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:04,132 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:04,132 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:11,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:15:11,239 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:11,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:14,142 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.32 % 2021-12-15 13:15:14,142 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 24.32 2021-12-15 13:15:14,157 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:14,161 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:14,161 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:21,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.80 % 2021-12-15 13:15:21,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:21,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:24,171 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.04 % 2021-12-15 13:15:24,171 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.04 2021-12-15 13:15:24,186 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:24,188 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:24,188 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:31,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:31,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:31,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:34,198 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:34,198 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:15:34,216 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:34,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:34,219 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:41,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:41,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:41,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:44,227 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:44,229 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:15:44,244 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:44,246 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:44,246 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:51,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:51,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:15:51,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:15:54,269 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:15:54,269 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:15:54,283 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:15:54,286 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:15:54,286 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:01,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:01,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:01,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:04,296 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:04,296 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:16:04,311 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:04,312 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:04,312 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:11,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:11,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:11,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:14,322 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:14,322 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:16:14,338 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:14,339 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:14,339 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:21,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:21,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:21,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:24,349 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:24,349 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:16:24,365 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:24,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:24,367 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:31,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:31,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:31,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:34,377 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.30 % 2021-12-15 13:16:34,377 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.3 2021-12-15 13:16:34,391 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:34,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:34,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:41,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.54 % 2021-12-15 13:16:41,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:41,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:44,404 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.54 % 2021-12-15 13:16:44,404 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.54 2021-12-15 13:16:44,420 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:44,423 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:44,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:51,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.79 % 2021-12-15 13:16:51,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:16:51,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:16:54,433 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.79 % 2021-12-15 13:16:54,433 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 25.79 2021-12-15 13:16:54,447 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:16:54,448 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:16:54,450 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:01,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.03 % 2021-12-15 13:17:01,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:01,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:04,460 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.03 % 2021-12-15 13:17:04,460 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.03 2021-12-15 13:17:04,476 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:04,479 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:04,479 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:11,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:11,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:11,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:14,487 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:14,489 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:17:14,503 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:14,506 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:14,506 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:21,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:21,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:21,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:24,516 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:24,516 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:17:24,532 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:24,535 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:24,535 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:31,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:31,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:31,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:34,548 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:34,548 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:17:34,565 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:34,569 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:34,569 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:41,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:41,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:41,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:44,579 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:44,581 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:17:44,598 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:44,602 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:44,602 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:51,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:51,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:17:51,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:17:54,614 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:17:54,615 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:17:54,644 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:17:54,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:17:54,648 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:01,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:18:01,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:01,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:04,658 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.27 % 2021-12-15 13:18:04,658 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.27 2021-12-15 13:18:04,677 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:04,678 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:04,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:11,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.53 % 2021-12-15 13:18:11,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:11,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:14,690 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.53 % 2021-12-15 13:18:14,690 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.53 2021-12-15 13:18:14,706 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:14,707 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:14,707 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:21,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.53 % 2021-12-15 13:18:21,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:21,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:24,720 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.53 % 2021-12-15 13:18:24,720 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 26.53 2021-12-15 13:18:24,736 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:24,739 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:24,740 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:31,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.01 % 2021-12-15 13:18:31,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:31,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:34,753 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.01 % 2021-12-15 13:18:34,753 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.01 2021-12-15 13:18:34,770 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:34,773 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:34,773 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:41,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:18:41,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:41,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:44,786 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:18:44,786 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:18:44,799 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:44,802 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:44,802 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:51,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:18:51,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:18:51,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:18:54,815 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:18:54,815 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:18:54,832 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:18:54,835 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:18:54,835 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:01,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:01,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:01,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:04,846 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:04,846 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:19:04,864 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:04,865 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:04,865 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:11,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:11,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:11,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:14,877 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:14,877 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:19:14,891 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:14,892 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:14,892 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:21,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:21,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:21,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:24,903 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:24,903 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:19:24,917 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:24,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:24,920 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:31,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:31,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:31,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:34,930 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.26 % 2021-12-15 13:19:34,930 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.26 2021-12-15 13:19:34,947 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:34,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:34,950 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:41,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.50 % 2021-12-15 13:19:41,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:41,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:44,961 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.50 % 2021-12-15 13:19:44,961 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.5 2021-12-15 13:19:44,980 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:44,982 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:44,982 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:51,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.50 % 2021-12-15 13:19:51,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:19:51,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:19:54,990 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.50 % 2021-12-15 13:19:54,990 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.5 2021-12-15 13:19:55,006 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:19:55,009 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:19:55,009 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:01,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.75 % 2021-12-15 13:20:01,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:01,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:05,023 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.75 % 2021-12-15 13:20:05,023 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 27.75 2021-12-15 13:20:05,042 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:05,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:05,046 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:11,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-15 13:20:11,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:11,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:15,056 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:15,056 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:20:15,073 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:15,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:15,076 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:21,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:21,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:21,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:25,089 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:25,089 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:20:25,107 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:25,109 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:25,109 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:31,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:31,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:31,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:35,119 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:35,121 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:20:35,140 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:35,142 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:35,142 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:41,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:41,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:41,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:45,153 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:45,154 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:20:45,168 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:45,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:45,171 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:51,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:51,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:20:51,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:20:55,186 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:20:55,186 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:20:55,200 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:20:55,203 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:20:55,203 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:01,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:21:01,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:01,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:05,216 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.22 % 2021-12-15 13:21:05,217 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.22 2021-12-15 13:21:05,230 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:05,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:05,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:11,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.47 % 2021-12-15 13:21:11,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:11,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:15,243 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.47 % 2021-12-15 13:21:15,243 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.47 2021-12-15 13:21:15,259 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:15,260 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:15,260 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:21,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.47 % 2021-12-15 13:21:21,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:21,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:25,273 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.71 % 2021-12-15 13:21:25,273 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.71 2021-12-15 13:21:25,289 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:25,292 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:25,293 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:31,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.71 % 2021-12-15 13:21:31,999 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:31,999 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:35,305 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.71 % 2021-12-15 13:21:35,305 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.71 2021-12-15 13:21:35,319 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:35,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:35,322 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:42,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.71 % 2021-12-15 13:21:42,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:42,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:45,331 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.71 % 2021-12-15 13:21:45,332 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 28.71 2021-12-15 13:21:45,345 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:45,348 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:45,348 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:52,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:21:52,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:21:52,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:21:55,358 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:21:55,358 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.21 2021-12-15 13:21:55,372 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:21:55,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:21:55,375 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:02,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:02,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:02,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:05,384 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:05,384 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.21 2021-12-15 13:22:05,400 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:05,403 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:05,403 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:12,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:12,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:12,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:15,413 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:15,413 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.21 2021-12-15 13:22:15,427 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:15,430 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:15,430 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:22,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:22,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:22,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:25,440 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.21 % 2021-12-15 13:22:25,440 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.21 2021-12-15 13:22:25,454 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:25,457 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:25,457 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:32,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.46 % 2021-12-15 13:22:32,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:32,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:35,467 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.46 % 2021-12-15 13:22:35,467 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.46 2021-12-15 13:22:35,483 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:35,483 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:35,484 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:42,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.46 % 2021-12-15 13:22:42,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:42,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:45,493 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.46 % 2021-12-15 13:22:45,494 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.46 2021-12-15 13:22:45,507 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:45,510 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:45,510 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:52,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.46 % 2021-12-15 13:22:52,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:22:52,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:22:55,520 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:22:55,522 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.69 2021-12-15 13:22:55,536 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:22:55,540 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:22:55,540 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:02,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:02,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:02,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:05,553 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:05,553 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.69 2021-12-15 13:23:05,569 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:05,572 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:05,573 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:12,193 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:12,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:12,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:15,585 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:15,585 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.69 2021-12-15 13:23:15,612 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:15,617 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:15,617 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:22,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:22,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:22,219 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:25,627 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:25,628 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.69 2021-12-15 13:23:25,641 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:25,644 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:25,644 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:32,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:32,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:32,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:35,658 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.69 % 2021-12-15 13:23:35,660 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 29.69 2021-12-15 13:23:35,677 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:35,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:35,680 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:42,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.19 % 2021-12-15 13:23:42,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:42,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:45,690 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.19 % 2021-12-15 13:23:45,691 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.19 2021-12-15 13:23:45,707 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:45,710 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:45,710 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:52,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.19 % 2021-12-15 13:23:52,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:23:52,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:23:55,720 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.19 % 2021-12-15 13:23:55,720 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.19 2021-12-15 13:23:55,734 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:23:55,736 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:23:55,736 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:02,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.44 % 2021-12-15 13:24:02,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:02,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:05,750 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:05,752 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:05,767 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:05,770 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:05,770 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:12,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:12,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:12,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:15,780 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:15,780 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:15,796 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:15,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:15,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:22,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:22,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:22,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:25,808 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:25,809 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:25,825 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:25,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:25,828 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:32,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:32,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:32,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:35,838 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:35,838 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:35,868 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:35,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:35,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:42,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:42,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:42,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:45,884 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:45,884 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:45,898 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:45,900 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:45,900 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:52,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:52,400 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:24:52,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:24:55,910 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.67 % 2021-12-15 13:24:55,910 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 30.67 2021-12-15 13:24:55,924 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:24:55,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:24:55,926 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:02,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.92 % 2021-12-15 13:25:02,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:02,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:05,936 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:05,937 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.17 2021-12-15 13:25:05,956 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:05,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:05,957 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:12,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:12,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:12,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:15,969 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:15,969 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.17 2021-12-15 13:25:15,987 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:15,989 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:15,990 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:22,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:22,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:22,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:26,000 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:26,000 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.17 2021-12-15 13:25:26,015 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:26,016 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:26,016 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:32,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.17 % 2021-12-15 13:25:32,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:32,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:36,029 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-15 13:25:36,029 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.42 2021-12-15 13:25:36,048 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:36,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:36,051 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:42,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-15 13:25:42,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:42,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:46,061 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-15 13:25:46,062 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.42 2021-12-15 13:25:46,075 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:46,078 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:46,078 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:52,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:25:52,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:25:52,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:25:56,091 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:25:56,091 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:25:56,108 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:25:56,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:25:56,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:02,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:02,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:02,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:06,121 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:06,121 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:26:06,135 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:06,138 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:06,138 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:12,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:12,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:12,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:16,148 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:16,148 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:26:16,165 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:16,167 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:16,167 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:22,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:22,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:22,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:26,180 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:26,180 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:26:26,198 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:26,201 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:26,201 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:32,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:32,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:32,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:36,211 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:36,213 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:26:36,230 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:36,232 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:36,233 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:42,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:42,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:42,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:46,243 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.66 % 2021-12-15 13:26:46,243 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 31.66 2021-12-15 13:26:46,259 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:46,263 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:46,263 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:52,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.90 % 2021-12-15 13:26:52,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:26:52,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:26:56,273 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.40 % 2021-12-15 13:26:56,273 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.4 2021-12-15 13:26:56,289 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:26:56,290 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:26:56,292 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:02,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.40 % 2021-12-15 13:27:02,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:02,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:06,302 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:06,303 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:06,322 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:06,323 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:06,325 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:12,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:12,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:12,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:16,335 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:16,335 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:16,354 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:16,357 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:16,357 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:22,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:22,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:22,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:26,377 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:26,377 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:26,398 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:26,403 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:26,403 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:32,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:32,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:32,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:36,414 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:36,414 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:36,428 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:36,431 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:36,431 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:42,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:42,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:42,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:46,443 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:46,443 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:46,460 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:46,463 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:46,463 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:52,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:52,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:27:52,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:27:56,473 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:27:56,474 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:27:56,490 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:27:56,492 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:27:56,492 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:02,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:28:02,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:02,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:06,502 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.64 % 2021-12-15 13:28:06,503 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.64 2021-12-15 13:28:06,516 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:06,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:06,519 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:12,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 13:28:12,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:12,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:16,529 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 13:28:16,529 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 13:28:16,546 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:16,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:16,549 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:22,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 13:28:22,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:22,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:26,562 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.12 % 2021-12-15 13:28:26,562 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.12 2021-12-15 13:28:26,585 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:26,591 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:26,591 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:32,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.12 % 2021-12-15 13:28:32,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:32,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:36,601 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.12 % 2021-12-15 13:28:36,601 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.12 2021-12-15 13:28:36,618 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:36,621 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:36,621 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:42,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:28:42,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:42,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:46,635 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:28:46,637 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:28:46,651 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:46,654 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:46,654 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:52,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:28:52,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:28:52,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:28:56,667 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:28:56,667 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:28:56,684 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:28:56,687 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:28:56,687 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:02,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:02,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:02,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:06,697 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:06,697 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:29:06,711 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:06,714 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:06,714 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:12,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:12,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:12,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:16,724 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:16,724 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:29:16,740 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:16,742 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:16,742 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:22,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:22,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:22,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:26,752 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:26,752 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:29:26,766 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:26,769 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:26,769 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:32,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:32,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:32,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:36,779 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:36,779 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:29:36,795 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:36,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:36,798 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:42,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:42,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:42,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:46,809 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.61 % 2021-12-15 13:29:46,809 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.61 2021-12-15 13:29:46,823 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:46,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:46,825 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:52,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.85 % 2021-12-15 13:29:52,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:29:52,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:29:56,835 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.85 % 2021-12-15 13:29:56,836 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 33.85 2021-12-15 13:29:56,851 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:29:56,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:29:56,855 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:02,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-15 13:30:02,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:02,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:06,867 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:06,867 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:06,882 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:06,885 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:06,885 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:13,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:13,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:13,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:16,897 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:16,898 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:16,911 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:16,914 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:16,914 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:23,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:23,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:23,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:26,924 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:26,924 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:26,940 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:26,941 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:26,941 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:33,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:33,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:33,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:36,953 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:36,954 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:36,970 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:36,973 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:36,973 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:43,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:43,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:43,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:46,983 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:46,983 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:46,996 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:46,999 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:46,999 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:53,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:53,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:30:53,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:30:57,009 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:30:57,009 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:30:57,026 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:30:57,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:30:57,028 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:03,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:31:03,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:03,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:07,039 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:31:07,039 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:31:07,055 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:07,055 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:07,056 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:13,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:31:13,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:13,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:17,066 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.60 % 2021-12-15 13:31:17,068 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.6 2021-12-15 13:31:17,082 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:17,085 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:17,085 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:23,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.84 % 2021-12-15 13:31:23,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:23,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:27,095 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.84 % 2021-12-15 13:31:27,095 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 34.84 2021-12-15 13:31:27,109 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:27,111 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:27,112 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:33,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.84 % 2021-12-15 13:31:33,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:33,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:37,122 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.10 % 2021-12-15 13:31:37,122 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.1 2021-12-15 13:31:37,141 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:37,144 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:37,144 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:43,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.10 % 2021-12-15 13:31:43,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:43,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:47,157 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-15 13:31:47,157 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.34 2021-12-15 13:31:47,177 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:47,180 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:47,180 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:53,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:31:53,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:31:53,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:31:57,193 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:31:57,194 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-15 13:31:57,209 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:31:57,210 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:31:57,211 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:03,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:03,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:03,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:07,221 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:07,223 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-15 13:32:07,239 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:07,242 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:07,242 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:13,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:13,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:13,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:17,253 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:17,253 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-15 13:32:17,269 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:17,272 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:17,272 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:23,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:23,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:23,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:27,283 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:27,285 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-15 13:32:27,302 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:27,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:27,305 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:33,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:33,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:33,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:37,315 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-15 13:32:37,315 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-15 13:32:37,328 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:37,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:37,332 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:43,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:32:43,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:43,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:47,342 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:32:47,344 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 13:32:47,358 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:47,361 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:47,361 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:53,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:32:53,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:32:53,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:32:57,374 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:32:57,374 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 13:32:57,391 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:32:57,392 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:32:57,394 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:03,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:33:03,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:03,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:07,403 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 13:33:07,404 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 13:33:07,418 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:07,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:07,421 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:13,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.09 % 2021-12-15 13:33:13,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:13,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:17,433 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.09 % 2021-12-15 13:33:17,433 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.09 2021-12-15 13:33:17,448 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:17,451 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:17,451 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:23,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.33 % 2021-12-15 13:33:23,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:23,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:27,461 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:27,461 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.58 2021-12-15 13:33:27,477 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:27,480 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:27,480 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:33,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:33,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:33,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:37,490 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:37,490 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.58 2021-12-15 13:33:37,506 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:37,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:37,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:43,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:43,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:43,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:47,519 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:47,519 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.58 2021-12-15 13:33:47,533 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:47,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:47,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:53,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:53,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:33:53,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:33:57,548 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:33:57,548 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.58 2021-12-15 13:33:57,563 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:33:57,566 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:33:57,566 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:03,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:34:03,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:03,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:07,576 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.58 % 2021-12-15 13:34:07,576 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.58 2021-12-15 13:34:07,605 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:07,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:07,608 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:13,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:13,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:13,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:17,619 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:17,621 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.83 2021-12-15 13:34:17,638 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:17,640 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:17,640 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:23,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:23,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:23,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:27,651 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:27,651 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.83 2021-12-15 13:34:27,667 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:27,671 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:27,671 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:33,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:33,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:33,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:37,680 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:37,680 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.83 2021-12-15 13:34:37,696 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:37,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:37,697 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:43,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:43,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:43,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:47,707 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.83 % 2021-12-15 13:34:47,707 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 36.83 2021-12-15 13:34:47,723 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:47,724 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:47,724 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:53,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-15 13:34:53,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:34:53,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:34:57,734 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-15 13:34:57,734 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.07 2021-12-15 13:34:57,752 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:34:57,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:34:57,753 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:03,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-15 13:35:03,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:03,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:07,765 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.31 % 2021-12-15 13:35:07,765 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.31 2021-12-15 13:35:07,780 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:07,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:07,782 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:13,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.56 % 2021-12-15 13:35:13,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:13,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:17,793 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.56 % 2021-12-15 13:35:17,795 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.56 2021-12-15 13:35:17,811 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:17,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:17,813 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:23,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:23,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:23,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:27,823 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:27,825 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:35:27,839 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:27,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:27,842 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:33,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:33,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:33,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:37,854 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:37,854 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:35:37,868 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:37,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:37,871 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:43,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:43,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:43,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:47,882 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:47,884 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:35:47,900 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:47,903 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:47,903 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:53,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:53,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:35:53,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:35:57,913 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:35:57,914 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:35:57,927 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:35:57,930 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:35:57,930 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:03,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:36:03,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:03,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:07,941 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:36:07,941 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:36:07,957 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:07,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:07,960 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:13,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:36:13,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:13,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:17,986 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 13:36:17,986 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 13:36:18,006 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:18,007 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:18,007 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:23,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.06 % 2021-12-15 13:36:23,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:23,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:28,019 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.06 % 2021-12-15 13:36:28,020 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.06 2021-12-15 13:36:28,036 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:28,036 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:28,038 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:33,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.06 % 2021-12-15 13:36:33,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:33,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:38,048 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.06 % 2021-12-15 13:36:38,049 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.06 2021-12-15 13:36:38,068 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:38,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:38,069 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:43,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.30 % 2021-12-15 13:36:43,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:43,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:48,079 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.54 % 2021-12-15 13:36:48,079 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.54 2021-12-15 13:36:48,092 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:48,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:48,095 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:53,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.54 % 2021-12-15 13:36:53,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:36:53,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:36:58,105 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:36:58,105 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.79 2021-12-15 13:36:58,122 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:36:58,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:36:58,125 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:03,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:03,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:03,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:08,137 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:08,137 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.79 2021-12-15 13:37:08,151 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:08,153 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:08,153 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:13,857 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:13,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:13,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:18,170 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:18,171 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.79 2021-12-15 13:37:18,193 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:18,196 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:18,196 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:23,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:23,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:23,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:28,206 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:28,207 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 38.79 2021-12-15 13:37:28,221 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:28,224 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:28,224 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:33,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.79 % 2021-12-15 13:37:33,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:33,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:38,234 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:37:38,234 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.04 2021-12-15 13:37:38,247 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:38,250 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:38,250 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:43,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:37:43,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:43,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:48,260 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:37:48,262 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.04 2021-12-15 13:37:48,276 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:48,278 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:48,279 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:53,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:37:53,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:37:53,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:37:58,289 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:37:58,289 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.04 2021-12-15 13:37:58,305 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:37:58,308 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:37:58,308 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:03,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-15 13:38:03,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:03,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:08,319 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.28 % 2021-12-15 13:38:08,319 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.28 2021-12-15 13:38:08,334 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:08,335 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:08,335 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:13,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.28 % 2021-12-15 13:38:13,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:13,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:18,345 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.28 % 2021-12-15 13:38:18,346 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.28 2021-12-15 13:38:18,359 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:18,362 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:18,362 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:23,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.28 % 2021-12-15 13:38:23,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:23,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:28,374 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.28 % 2021-12-15 13:38:28,374 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.28 2021-12-15 13:38:28,391 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:28,395 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:28,395 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:34,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.53 % 2021-12-15 13:38:34,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:34,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:38,407 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.53 % 2021-12-15 13:38:38,407 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.53 2021-12-15 13:38:38,421 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:38,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:38,424 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:44,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:38:44,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:44,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:48,434 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:38:48,434 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.77 2021-12-15 13:38:48,453 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:48,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:48,454 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:54,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:38:54,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:38:54,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:38:58,464 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:38:58,464 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.77 2021-12-15 13:38:58,480 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:38:58,483 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:38:58,483 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:04,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:39:04,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:39:04,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:08,493 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:39:08,493 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 39.77 2021-12-15 13:39:08,507 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:39:08,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:39:08,509 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:14,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.77 % 2021-12-15 13:39:14,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:39:14,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:18,519 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-15 13:39:18,519 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 40.25 2021-12-15 13:39:18,533 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:39:18,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:39:18,536 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:24,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-15 13:39:24,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:39:24,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:28,546 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-15 13:39:28,546 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 40.25 2021-12-15 13:39:28,561 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:39:28,562 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:39:28,562 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:34,098 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.25% in 3586.97000003 seconds 2021-12-15 13:39:34,109 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 13:39:34,109 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 13:39:34,111 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 13:39:34,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:39:34,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 13:39:34,493 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:39:34,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 13:39:34,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 13:39:34,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 13:39:34,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 13:39:34,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 13:39:34,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 13:39:34,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 13:39:34,598 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 13:39:34,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 13:39:34,660 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['109066 memcached\n'], []) 2021-12-15 13:39:34,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 13:39:34,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 13:39:34,750 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['109066 memcached\n'], []) 2021-12-15 13:39:34,750 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 13:39:34,755 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 13:39:34,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:39:35,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 13:39:35,088 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:39:35,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 13:39:35,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 13:39:35,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 13:39:35,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 13:39:35,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 13:39:35,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 13:39:35,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 13:39:35,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:39:35,226 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 13:39:35,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:39:35,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:35,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:35,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:35,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:35,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:35,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:35,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:35,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:35,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:35,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:35,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 13:39:35,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 13:39:35,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:35,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 13:39:35,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 13:39:35,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:39:35,892 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 13:39:35,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:39:35,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:35,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:39:35,956 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 13:39:35,956 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 13:39:35,959 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 13:39:36,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:39:36,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 13:39:36,520 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:39:36,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 13:39:36,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 13:39:36,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 13:39:36,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 13:39:36,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 13:39:36,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 13:39:36,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 13:39:36,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:39:36,694 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 13:39:36,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:39:36,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:36,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:36,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:36,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:36,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:36,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:37,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:37,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:37,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:37,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:37,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:37,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:37,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:39:37,223 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 13:39:37,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:39:37,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:37,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:39:37,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 13:39:37,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 13:39:37,279 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 13:39:37,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:39:37,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 13:39:37,744 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:39:37,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 13:39:37,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 13:39:37,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 13:39:37,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 13:39:37,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 13:39:37,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 13:39:37,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 13:39:37,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:39:37,868 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 13:39:37,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:39:37,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:37,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:37,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:37,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:37,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:39:37,993 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 13:39:37,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:39:38,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:38,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:39:38,032 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 13:39:38,032 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 13:39:38,036 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 13:39:38,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:39:38,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 13:39:38,575 | test | DEBUG | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-15 13:39:38,575 | test | INFO | pool-3-thread-21 | [task:check:385] Rebalance - status: running, progress: 40.25 2021-12-15 13:39:38,581 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:39:38,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 13:39:38,595 | infra | DEBUG | pool-3-thread-21 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:39:38,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:39:38,596 | infra | DEBUG | pool-3-thread-21 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:39:38,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 13:39:38,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 13:39:38,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 13:39:38,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 13:39:38,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 13:39:38,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 13:39:38,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:39:38,716 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 13:39:38,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:39:38,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:38,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:38,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:38,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:39:38,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:38,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:39,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:39,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:39:39,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:39,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:39:39,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:39,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:39:39,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:39:39,167 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 13:39:39,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:39:39,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:39:39,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:39:39,210 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 13:39:39,210 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 13:39:39,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 13:39:39,216 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 13:39:48,605 | test | ERROR | pool-3-thread-21 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'102187c03295ee9bc63a2c6d71bff60f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=510484cb33bbdce8670cc83da47e2887', u'status': u'notRunning'} - rebalance failed 2021-12-15 13:39:48,622 | test | INFO | pool-3-thread-21 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 13:39:48,622 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384068L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.068Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.17549.116>,\n {{{{nocatch,{error,closed}},\n [{mc_binary,recv_with_data,4,\n [{file,"src/mc_binary.erl"},{line,41}]},\n {mc_binary,quick_active_recv,3,\n [{file,"src/mc_binary.erl"},{line,48}]},\n {mc_binary,quick_stats_loop_enter,5,\n [{file,"src/mc_binary.erl"},{line,100}]},\n {mc_binary,quick_stats,5,\n [{file,"src/mc_binary.erl"},{line,85}]},\n {mc_client_binary,get_dcp_docs_estimate,\n 3,\n [{file,"src/mc_client_binary.erl"},\n {line,754}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,616}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,121,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.18323.110>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 291}},\n infinity]}}}}}.\nRebalance Operation Id = d19fc0aba12ea674050f5f5a0a03f06f'} 2021-12-15 13:39:48,624 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384059L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.059Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 12:31:49.944420 109296 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 13:39:48,624 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384034L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.034Z', u'text': u'Worker <0.17581.116> (for action {move,\n {291,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.17549.116>,\n {{{{nocatch,\n {error,\n closed}},\n [{mc_binary,\n recv_with_data,\n 4,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 41}]},\n {mc_binary,\n quick_active_recv,\n 3,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 48}]},\n {mc_binary,\n quick_stats_loop_enter,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 100}]},\n {mc_binary,\n quick_stats,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 85}]},\n {mc_client_binary,\n get_dcp_docs_estimate,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 754}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 616}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 225}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 121,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.18323.110>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 291}},\n infinity]}}}}'} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384021L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.021Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600667542L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:37:47.542Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600667355L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:37:47.355Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600667137L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:37:47.137Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d19fc0aba12ea674050f5f5a0a03f06f"} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600658159L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:37:38.159Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 321 seconds.'} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301264L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.264Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nW1215 05:16:06.940414 75546 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 06:37:23.493861 33679 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 08:12:20.866575 33677 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 10:17:14.209003 33682 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object\nW1215 11:19:57.221807 33681 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639600301248L, u'shortText': u'message', u'serverTime': u'2021-12-15T12:31:41.248Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30088.109>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,506}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {set_vbucket,151,active,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.8660.106>,\n {update_vbucket_state,151,active,paused,\n undefined,\n [[\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\']]}},\n infinity]}}}}}.\nRebalance Operation Id = 274721e656eeff3e99e26fb4ac58d12d'} 2021-12-15 13:39:48,625 | test | ERROR | pool-3-thread-21 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'102187c03295ee9bc63a2c6d71bff60f', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=510484cb33bbdce8670cc83da47e2887', u'status': u'notRunning'} - rebalance failed 2021-12-15 13:40:39,230 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 13:40:39,230 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 13:40:39,243 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:39,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:39,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:41,269 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:41,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:41,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:43,289 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:43,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:43,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:45,312 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:45,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:45,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:47,334 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:47,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:47,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:49,352 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:49,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:49,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:51,377 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:51,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:51,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:53,401 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:53,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:53,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:55,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:55,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:55,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:57,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:57,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:57,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:40:59,459 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:40:59,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:40:59,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:01,479 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:01,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:01,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:03,499 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:03,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:03,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:05,522 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:05,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:05,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:07,540 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:07,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:07,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:09,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:09,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:09,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:11,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:11,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:11,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:13,595 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:13,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:13,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:15,615 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:15,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:15,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:17,637 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:17,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:17,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:19,657 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:19,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:19,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:21,678 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:21,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:21,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:23,701 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:23,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:23,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:25,719 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:25,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:25,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:27,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:27,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:27,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:32,371 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:32,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:32,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:34,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:34,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:34,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:36,407 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:36,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:36,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:38,424 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:38,433 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:38,433 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:40,448 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:40,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:40,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:42,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:42,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:42,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:44,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:44,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:44,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:46,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:46,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:46,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:48,538 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:48,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:48,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:50,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:50,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:50,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:52,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:52,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:52,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:54,598 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:54,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:54,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:56,621 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:56,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:56,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:41:58,644 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:41:58,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:41:58,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:00,665 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:00,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:00,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:02,686 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:02,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:02,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:04,703 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:04,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:04,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:06,726 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:06,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:06,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:08,747 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:08,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:08,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:10,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:10,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:10,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:12,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:12,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:12,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:14,819 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:14,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:14,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:16,838 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:16,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:16,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:18,858 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:18,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:18,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:20,887 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:20,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:20,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:22,907 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:22,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:22,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:27,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:27,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:27,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:29,414 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:29,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:29,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:31,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:31,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:31,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:33,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:33,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:33,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:35,473 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:35,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:35,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:37,494 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:37,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:37,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:39,513 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:39,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:39,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:41,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:41,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:41,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:43,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:43,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:43,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:45,579 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:45,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:45,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:47,599 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:47,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:47,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:49,621 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:49,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:49,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:51,640 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:51,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:51,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:53,663 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:53,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:53,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:55,687 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:55,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:55,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:57,707 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:57,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:57,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:42:59,726 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:42:59,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:42:59,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:01,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:01,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:01,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:03,770 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:03,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:03,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:05,792 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:05,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:05,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:07,813 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:07,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:07,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:09,832 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:09,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:09,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:11,851 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:11,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:11,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:13,874 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:13,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:13,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:15,895 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:15,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:15,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:17,918 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:17,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:17,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:19,938 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:19,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:19,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:21,957 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:21,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:21,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:23,979 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:23,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:23,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:25,999 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:26,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:26,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:28,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:28,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:28,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:30,046 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:30,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:30,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:32,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:32,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:32,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:34,089 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:34,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:34,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:36,109 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:36,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:36,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:38,130 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:38,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:38,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:40,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:40,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:40,155 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:42,170 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:42,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:42,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:44,188 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:44,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:44,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:46,210 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:46,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:46,219 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:48,234 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:48,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:48,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:50,256 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:50,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:50,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:52,276 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:52,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:52,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:54,296 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:54,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:54,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:56,318 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:56,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:56,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:43:58,341 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:43:58,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:43:58,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:00,364 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:00,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:00,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:02,388 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:02,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:02,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:04,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:04,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:04,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:06,436 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:06,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:06,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:08,459 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:08,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:08,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:10,480 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:10,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:10,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:12,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:12,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:12,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:14,528 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:14,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:14,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:16,552 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:16,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:16,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:18,571 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:18,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:18,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:20,591 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:20,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:20,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:22,609 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:22,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:22,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:24,631 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:24,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:24,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:26,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:26,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:26,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:28,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:28,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:28,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:30,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:30,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:30,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:32,717 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:32,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:32,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:34,736 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:34,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:34,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:36,756 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:36,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:36,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:38,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:38,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:38,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:40,796 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:40,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:40,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:42,815 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:42,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:42,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:44,835 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:44,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:44,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:46,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:46,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:46,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:48,882 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:48,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:48,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:50,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:50,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:50,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:52,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:52,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:52,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:54,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:54,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:54,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:56,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 13:44:56,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 13:44:56,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 13:44:58,984 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 13:44:58,999 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 13:44:58,999 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 13:44:59,009 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 13:44:59,023 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-15 13:44:59,023 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 13:44:59,035 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-15 13:44:59,046 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 13:44:59,046 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 13:44:59,059 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 13:44:59,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 13:44:59,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:44:59,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 13:44:59,507 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:44:59,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 13:44:59,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 13:44:59,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 13:44:59,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 13:44:59,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 13:44:59,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 13:44:59,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 13:44:59,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:44:59,683 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 13:44:59,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:44:59,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:44:59,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:44:59,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:44:59,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:44:59,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:44:59,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:44:59,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:44:59,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:44:59,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:44:59,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:44:59,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:00,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:00,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:00,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:00,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 13:45:00,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 13:45:00,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:00,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:00,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 13:45:00,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 13:45:00,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:45:00,210 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 13:45:00,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:45:00,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:00,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:45:00,253 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 13:45:00,253 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 13:45:00,256 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 13:45:00,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:45:00,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 13:45:00,703 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:45:00,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 13:45:00,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 13:45:00,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 13:45:00,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 13:45:00,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 13:45:00,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 13:45:00,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 13:45:00,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:45:00,813 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 13:45:00,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:45:00,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:00,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:00,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:00,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:00,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:00,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:01,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:45:01,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:45:01,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:01,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:01,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:01,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:01,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:45:01,148 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 13:45:01,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:45:01,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:01,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:45:01,180 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 13:45:01,180 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 13:45:01,184 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 13:45:01,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:45:01,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 13:45:01,582 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:45:01,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 13:45:01,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 13:45:01,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 13:45:01,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 13:45:01,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 13:45:01,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 13:45:01,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 13:45:01,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:45:01,691 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 13:45:01,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:45:01,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:01,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:01,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:01,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:01,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:45:01,815 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 13:45:01,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:45:01,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:01,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:45:01,851 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 13:45:01,851 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 13:45:01,855 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 13:45:01,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 13:45:02,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 13:45:02,292 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 13:45:02,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 13:45:02,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 13:45:02,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 13:45:02,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 13:45:02,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 13:45:02,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 13:45:02,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 13:45:02,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 13:45:02,421 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 13:45:02,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 13:45:02,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:02,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:02,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:02,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 13:45:02,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:02,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:02,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:45:02,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 13:45:02,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:02,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 13:45:02,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:02,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 13:45:02,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 13:45:02,799 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 13:45:02,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 13:45:02,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 13:45:02,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 13:45:02,838 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 13:45:02,838 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 13:45:02,838 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1639600667.08 2021-12-15 13:45:02,838 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 40 2021-12-15 13:45:02,871 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1639604702.84 2021-12-15 13:45:02,874 | infra | INFO | pool-3-thread-22 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[]_1639604702.84' started 2021-12-15 13:45:02,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 13:45:02,881 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 13:45:02,900 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-15 13:45:02,956 | test | DEBUG | pool-3-thread-22 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 13:45:02,960 | test | INFO | pool-3-thread-22 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 5.78785764131 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 3.15793864894 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 2.07049132425 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 2.33302263463 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 13:45:02,971 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:02,971 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:02,993 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:02,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:02,996 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:45:13,010 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:13,012 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:13,028 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:13,030 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:13,032 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:45:23,046 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:23,046 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:23,062 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:23,063 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:23,065 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:45:33,076 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:33,076 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:33,094 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:33,096 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:33,096 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:45:43,115 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:43,117 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:43,131 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:43,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:43,134 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:45:53,144 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:45:53,145 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:45:53,158 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:45:53,161 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:45:53,161 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:03,171 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 13:46:03,173 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 13:46:03,186 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:03,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:03,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:13,200 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.47 % 2021-12-15 13:46:13,200 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.47 2021-12-15 13:46:13,217 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:13,219 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:13,219 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:23,230 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.83 % 2021-12-15 13:46:23,230 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.83 2021-12-15 13:46:23,243 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:23,246 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:23,246 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:33,262 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.83 % 2021-12-15 13:46:33,263 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.83 2021-12-15 13:46:33,282 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:33,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:33,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:43,298 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.83 % 2021-12-15 13:46:43,298 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 0.83 2021-12-15 13:46:43,319 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:43,321 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:43,321 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:46:53,332 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.29 % 2021-12-15 13:46:53,332 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 1.29 2021-12-15 13:46:53,345 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:46:53,346 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:46:53,348 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:02,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:02,897 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 1.71 2021-12-15 13:47:02,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:02,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:02,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:03,357 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:03,357 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 1.71 2021-12-15 13:47:03,371 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:03,372 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:03,372 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:12,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:12,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:12,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:13,384 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:13,385 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 1.71 2021-12-15 13:47:13,401 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:13,404 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:13,404 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:22,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:22,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:22,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:23,414 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.71 % 2021-12-15 13:47:23,415 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 1.71 2021-12-15 13:47:23,430 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:23,431 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:23,431 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:32,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.12 % 2021-12-15 13:47:32,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:32,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:33,443 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.12 % 2021-12-15 13:47:33,443 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.12 2021-12-15 13:47:33,459 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:33,460 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:33,460 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:43,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:47:43,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:43,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:43,473 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:47:43,473 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-15 13:47:43,493 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:43,494 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:43,494 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:53,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:47:53,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:47:53,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:47:53,506 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:47:53,506 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-15 13:47:53,526 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:47:53,528 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:47:53,529 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:03,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:48:03,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:03,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:03,539 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-15 13:48:03,539 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-15 13:48:03,553 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:03,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:03,555 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:13,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:13,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:13,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:13,568 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:13,568 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.95 2021-12-15 13:48:13,588 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:13,591 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:13,591 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:23,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:23,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:23,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:23,601 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:23,601 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.95 2021-12-15 13:48:23,618 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:23,619 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:23,619 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:33,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:33,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:33,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:33,631 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.95 % 2021-12-15 13:48:33,631 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 2.95 2021-12-15 13:48:33,651 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:33,654 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:33,654 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:43,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.42 % 2021-12-15 13:48:43,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:43,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:43,664 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.42 % 2021-12-15 13:48:43,664 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 3.42 2021-12-15 13:48:43,678 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:43,681 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:43,681 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:53,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:48:53,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:48:53,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:48:53,693 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:48:53,693 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 3.84 2021-12-15 13:48:53,710 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:48:53,713 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:48:53,713 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:03,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:49:03,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:03,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:03,723 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:49:03,723 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 3.84 2021-12-15 13:49:03,740 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:03,743 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:03,743 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:13,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:49:13,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:13,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:13,756 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.84 % 2021-12-15 13:49:13,757 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 3.84 2021-12-15 13:49:13,775 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:13,778 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:13,778 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:23,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 13:49:23,244 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:23,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:23,792 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 13:49:23,793 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 4.2 2021-12-15 13:49:23,812 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:23,816 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:23,816 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:33,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:33,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:33,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:33,831 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:33,831 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 4.61 2021-12-15 13:49:33,845 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:33,849 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:33,849 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:43,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:43,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:43,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:43,869 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:43,869 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 4.61 2021-12-15 13:49:43,887 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:43,891 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:43,891 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:53,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:53,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:49:53,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:49:53,901 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.61 % 2021-12-15 13:49:53,901 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 4.61 2021-12-15 13:49:53,918 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:49:53,921 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:49:53,921 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:03,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:03,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:03,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:03,933 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:03,933 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 5.07 2021-12-15 13:50:03,948 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:03,951 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:03,951 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:13,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:13,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:13,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:13,961 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:13,963 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 5.07 2021-12-15 13:50:13,976 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:13,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:13,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:23,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:23,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:23,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:23,990 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:23,990 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 5.07 2021-12-15 13:50:24,005 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:24,007 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:24,007 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:33,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:33,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:33,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:34,017 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-15 13:50:34,017 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 5.07 2021-12-15 13:50:34,033 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:34,036 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:34,036 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:43,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.90 % 2021-12-15 13:50:43,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:43,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:44,051 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.90 % 2021-12-15 13:50:44,052 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 5.9 2021-12-15 13:50:44,068 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:44,071 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:44,071 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:53,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:50:53,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:50:53,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:50:54,082 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:50:54,082 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 13:50:54,098 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:50:54,101 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:50:54,101 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:03,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:03,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:03,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:04,114 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:04,114 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 13:51:04,131 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:04,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:04,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:13,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:13,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:13,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:14,153 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:14,153 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 13:51:14,170 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:14,173 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:14,173 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:23,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:23,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:23,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:24,187 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:24,188 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 13:51:24,203 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:24,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:24,207 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:33,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:33,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:33,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:34,219 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 13:51:34,219 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 13:51:34,236 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:34,239 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:34,239 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:43,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.73 % 2021-12-15 13:51:43,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:43,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:44,252 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.73 % 2021-12-15 13:51:44,252 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.73 2021-12-15 13:51:44,270 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:44,273 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:44,273 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:53,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.73 % 2021-12-15 13:51:53,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:51:53,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:51:54,288 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.73 % 2021-12-15 13:51:54,288 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 6.73 2021-12-15 13:51:54,302 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:51:54,305 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:51:54,305 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:03,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.20 % 2021-12-15 13:52:03,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:03,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:04,313 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.20 % 2021-12-15 13:52:04,315 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 7.2 2021-12-15 13:52:04,328 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:04,329 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:04,329 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:13,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.66 % 2021-12-15 13:52:13,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:13,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:14,345 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.66 % 2021-12-15 13:52:14,345 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 7.66 2021-12-15 13:52:14,367 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:14,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:14,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:23,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.66 % 2021-12-15 13:52:23,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:23,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:24,378 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:24,378 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 13:52:24,391 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:24,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:24,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:33,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:33,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:33,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:34,407 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:34,407 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 13:52:34,427 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:34,428 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:34,428 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:43,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:43,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:43,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:44,440 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:44,440 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 13:52:44,453 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:44,456 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:44,456 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:53,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:53,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:52:53,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:52:54,467 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:52:54,469 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 13:52:54,486 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:52:54,489 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:52:54,489 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:03,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:53:03,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:03,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:04,502 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 13:53:04,502 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 13:53:04,516 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:04,517 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:04,517 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:13,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:13,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:13,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:14,529 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:14,530 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.45 2021-12-15 13:53:14,548 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:14,549 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:14,549 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:23,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:23,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:23,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:24,562 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:24,562 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.45 2021-12-15 13:53:24,576 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:24,578 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:24,578 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:33,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:33,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:33,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:34,588 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:34,588 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.45 2021-12-15 13:53:34,601 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:34,604 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:34,604 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:43,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:43,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:43,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:44,615 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.45 % 2021-12-15 13:53:44,615 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 8.45 2021-12-15 13:53:44,630 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:44,631 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:44,632 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:53,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:53:53,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:53:53,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:53:54,644 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:53:54,644 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 9.68 2021-12-15 13:53:54,665 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:53:54,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:53:54,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:03,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:03,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:03,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:04,688 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:04,688 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 9.68 2021-12-15 13:54:04,707 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:04,710 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:04,710 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:13,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:13,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:13,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:14,721 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:14,721 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 9.68 2021-12-15 13:54:14,740 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:14,743 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:14,743 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:23,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:23,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:23,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:24,753 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.68 % 2021-12-15 13:54:24,753 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 9.68 2021-12-15 13:54:24,769 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:24,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:24,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:33,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:33,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:33,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:34,782 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:34,782 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.15 2021-12-15 13:54:34,798 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:34,801 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:34,801 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:43,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:43,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:43,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:44,811 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:44,811 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.15 2021-12-15 13:54:44,825 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:44,826 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:44,826 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:53,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:53,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:54:53,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:54:54,838 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:54:54,838 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.15 2021-12-15 13:54:54,851 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:54:54,855 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:54:54,855 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:03,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:55:03,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:03,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:04,865 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-15 13:55:04,865 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.15 2021-12-15 13:55:04,882 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:04,885 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:04,885 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:13,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.57 % 2021-12-15 13:55:13,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:13,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:14,898 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.57 % 2021-12-15 13:55:14,898 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.57 2021-12-15 13:55:14,913 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:14,915 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:14,915 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:23,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.57 % 2021-12-15 13:55:23,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:23,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:24,926 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.57 % 2021-12-15 13:55:24,927 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 10.57 2021-12-15 13:55:24,953 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:24,954 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:24,954 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:33,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 13:55:34,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:34,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:34,964 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 13:55:34,964 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.34 2021-12-15 13:55:34,980 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:34,983 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:34,983 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:44,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 13:55:44,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:44,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:44,997 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 13:55:44,997 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.34 2021-12-15 13:55:45,015 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:45,017 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:45,017 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:54,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:55:54,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:55:54,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:55:55,029 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:55:55,029 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:55:55,045 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:55:55,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:55:55,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:04,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:04,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:04,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:05,059 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:05,059 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:05,073 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:05,076 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:05,076 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:14,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:14,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:14,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:15,088 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:15,088 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:15,105 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:15,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:15,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:24,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:24,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:24,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:25,119 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:25,119 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:25,134 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:25,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:25,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:34,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:34,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:34,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:35,147 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:35,148 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:35,163 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:35,165 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:35,167 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:44,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:44,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:44,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:45,180 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:45,180 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:45,201 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:45,204 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:45,204 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:54,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:54,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:56:54,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:56:55,217 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.81 % 2021-12-15 13:56:55,217 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 11.81 2021-12-15 13:56:55,232 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:56:55,234 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:56:55,234 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:04,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.27 % 2021-12-15 13:57:04,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:04,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:05,246 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.27 % 2021-12-15 13:57:05,246 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 12.27 2021-12-15 13:57:05,260 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:05,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:05,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:14,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:14,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:14,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:15,276 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:15,276 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:57:15,292 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:15,295 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:15,295 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:24,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:24,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:24,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:25,306 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:25,306 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:57:25,329 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:25,332 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:25,332 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:34,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:34,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:34,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:35,345 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:35,345 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:57:35,359 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:35,361 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:35,361 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:44,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:44,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:44,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:45,374 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:45,375 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:57:45,388 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:45,391 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:45,391 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:54,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:54,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:57:54,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:57:55,403 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:57:55,403 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:57:55,417 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:57:55,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:57:55,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:04,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:58:04,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:04,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:05,430 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:58:05,430 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.47 2021-12-15 13:58:05,446 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:05,447 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:05,447 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:14,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.47 % 2021-12-15 13:58:14,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:14,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:15,459 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.93 % 2021-12-15 13:58:15,459 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.93 2021-12-15 13:58:15,476 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:15,479 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:15,479 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:24,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.93 % 2021-12-15 13:58:24,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:24,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:25,490 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.93 % 2021-12-15 13:58:25,492 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.93 2021-12-15 13:58:25,513 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:25,515 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:25,515 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:34,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.93 % 2021-12-15 13:58:34,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:34,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:35,523 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.93 % 2021-12-15 13:58:35,525 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 13.93 2021-12-15 13:58:35,539 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:35,540 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:35,540 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:44,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.29 % 2021-12-15 13:58:44,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:44,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:45,553 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.29 % 2021-12-15 13:58:45,553 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 14.29 2021-12-15 13:58:45,573 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:45,575 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:45,575 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:54,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.76 % 2021-12-15 13:58:54,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:58:54,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:58:55,586 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.76 % 2021-12-15 13:58:55,586 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 14.76 2021-12-15 13:58:55,609 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:58:55,611 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:58:55,611 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:04,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:04,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:04,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:05,621 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:05,622 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.18 2021-12-15 13:59:05,641 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:05,644 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:05,644 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:14,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:14,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:14,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:15,653 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:15,653 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.18 2021-12-15 13:59:15,665 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:15,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:15,668 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:24,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:24,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:24,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:25,680 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 13:59:25,681 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.18 2021-12-15 13:59:25,700 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:25,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:25,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:34,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:34,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:34,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:35,710 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:35,711 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.59 2021-12-15 13:59:35,730 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:35,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:35,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:44,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:44,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:44,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:45,746 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:45,746 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.59 2021-12-15 13:59:45,763 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:45,766 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:45,766 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:54,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:54,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 13:59:54,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 13:59:55,779 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-15 13:59:55,779 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.59 2021-12-15 13:59:55,796 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 13:59:55,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 13:59:55,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:04,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.95 % 2021-12-15 14:00:04,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:04,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:05,809 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.95 % 2021-12-15 14:00:05,811 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.95 2021-12-15 14:00:05,832 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:05,835 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:05,835 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:14,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.95 % 2021-12-15 14:00:14,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:14,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:15,846 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.95 % 2021-12-15 14:00:15,846 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 15.95 2021-12-15 14:00:15,865 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:15,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:15,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:24,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.42 % 2021-12-15 14:00:24,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:24,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:25,885 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.42 % 2021-12-15 14:00:25,885 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 16.42 2021-12-15 14:00:25,903 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:25,905 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:25,907 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:34,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.88 % 2021-12-15 14:00:34,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:34,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:35,924 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.88 % 2021-12-15 14:00:35,924 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 16.88 2021-12-15 14:00:35,948 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:35,953 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:35,953 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:44,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.88 % 2021-12-15 14:00:44,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:44,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:45,967 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.88 % 2021-12-15 14:00:45,967 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 16.88 2021-12-15 14:00:45,986 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:45,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:45,990 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:54,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:00:54,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:00:54,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:00:56,003 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:00:56,003 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.3 2021-12-15 14:00:56,022 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:00:56,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:00:56,026 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:04,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:04,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:04,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:06,038 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:06,039 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.3 2021-12-15 14:01:06,055 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:06,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:06,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:14,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:14,683 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:14,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:16,071 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:16,071 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.3 2021-12-15 14:01:16,085 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:16,088 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:16,088 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:24,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:24,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:24,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:26,104 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:26,104 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.3 2021-12-15 14:01:26,122 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:26,125 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:26,125 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:34,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:34,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:34,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:36,138 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.30 % 2021-12-15 14:01:36,138 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.3 2021-12-15 14:01:36,154 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:36,155 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:36,155 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:44,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:01:44,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:44,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:46,170 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:01:46,170 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.67 2021-12-15 14:01:46,184 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:46,186 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:46,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:54,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:01:54,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:01:54,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:01:56,197 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:01:56,197 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.67 2021-12-15 14:01:56,217 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:01:56,220 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:01:56,220 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:04,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:02:04,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:04,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:06,233 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.67 % 2021-12-15 14:02:06,233 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 17.67 2021-12-15 14:02:06,250 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:06,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:06,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:14,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.07 % 2021-12-15 14:02:14,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:14,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:16,267 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.54 % 2021-12-15 14:02:16,267 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 18.54 2021-12-15 14:02:16,283 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:16,286 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:16,286 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:24,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:24,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:24,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:26,299 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:26,299 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:02:26,315 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:26,318 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:26,318 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:34,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:34,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:34,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:36,332 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:36,332 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:02:36,351 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:36,354 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:36,354 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:44,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:44,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:44,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:46,384 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:46,385 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:02:46,401 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:46,403 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:46,403 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:54,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:54,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:02:54,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:02:56,414 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:02:56,414 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:02:56,430 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:02:56,433 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:02:56,433 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:04,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:03:04,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:04,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:06,446 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:03:06,447 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:03:06,466 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:06,467 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:06,467 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:14,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:03:14,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:14,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:16,480 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.01 % 2021-12-15 14:03:16,480 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.01 2021-12-15 14:03:16,494 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:16,496 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:16,496 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:24,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-15 14:03:24,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:24,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:26,510 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.79 % 2021-12-15 14:03:26,510 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.79 2021-12-15 14:03:26,530 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:26,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:26,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:34,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.79 % 2021-12-15 14:03:34,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:34,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:36,545 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.79 % 2021-12-15 14:03:36,546 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.79 2021-12-15 14:03:36,566 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:36,569 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:36,569 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:44,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.79 % 2021-12-15 14:03:45,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:45,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:46,582 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.79 % 2021-12-15 14:03:46,582 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 19.79 2021-12-15 14:03:46,601 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:46,604 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:46,604 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:55,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.20 % 2021-12-15 14:03:55,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:03:55,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:03:56,618 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.20 % 2021-12-15 14:03:56,618 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 20.2 2021-12-15 14:03:56,638 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:03:56,640 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:03:56,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:05,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:05,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:05,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:06,651 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:06,651 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 20.66 2021-12-15 14:04:06,665 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:06,667 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:06,667 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:15,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:15,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:15,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:16,680 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:16,680 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 20.66 2021-12-15 14:04:16,700 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:16,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:16,703 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:25,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:25,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:25,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:26,714 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:26,714 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 20.66 2021-12-15 14:04:26,730 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:26,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:26,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:35,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:35,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:35,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:36,744 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:36,746 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 20.66 2021-12-15 14:04:36,765 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:36,767 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:36,767 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:45,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.66 % 2021-12-15 14:04:45,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:45,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:46,780 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.03 % 2021-12-15 14:04:46,780 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 21.03 2021-12-15 14:04:46,796 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:46,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:46,799 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:55,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:04:55,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:04:55,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:04:56,812 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:04:56,812 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 21.49 2021-12-15 14:04:56,828 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:04:56,831 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:04:56,831 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:05,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:05,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:05,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:06,842 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:06,842 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 21.49 2021-12-15 14:05:06,865 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:06,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:06,868 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:15,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:15,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:15,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:16,881 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:16,882 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 21.49 2021-12-15 14:05:16,904 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:16,907 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:16,907 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:25,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:25,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:25,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:26,921 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.49 % 2021-12-15 14:05:26,923 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 21.49 2021-12-15 14:05:26,938 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:26,943 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:26,943 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:35,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.91 % 2021-12-15 14:05:35,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:35,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:36,959 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.32 % 2021-12-15 14:05:36,960 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 22.32 2021-12-15 14:05:36,976 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:36,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:36,980 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:45,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.32 % 2021-12-15 14:05:45,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:45,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:46,993 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.32 % 2021-12-15 14:05:46,994 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 22.32 2021-12-15 14:05:47,012 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:47,015 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:47,015 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:55,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.32 % 2021-12-15 14:05:55,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:05:55,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:05:57,026 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.32 % 2021-12-15 14:05:57,026 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 22.32 2021-12-15 14:05:57,042 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:05:57,046 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:05:57,046 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:05,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.79 % 2021-12-15 14:06:05,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:05,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:07,059 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.79 % 2021-12-15 14:06:07,059 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 22.79 2021-12-15 14:06:07,075 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:07,078 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:07,078 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:15,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.79 % 2021-12-15 14:06:15,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:15,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:17,089 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.79 % 2021-12-15 14:06:17,091 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 22.79 2021-12-15 14:06:17,108 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:17,111 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:17,111 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:25,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.79 % 2021-12-15 14:06:25,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:25,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:27,122 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:27,124 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:06:27,153 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:27,155 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:27,155 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:35,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:35,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:35,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:37,167 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:37,167 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:06:37,184 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:37,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:37,187 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:45,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:45,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:45,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:47,198 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:47,198 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:06:47,213 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:47,216 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:47,217 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:55,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:55,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:06:55,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:06:57,229 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:06:57,229 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:06:57,259 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:06:57,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:06:57,263 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:05,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:07:05,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:05,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:07,273 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:07:07,273 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:07:07,289 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:07,290 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:07,292 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:15,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:07:15,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:15,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:17,302 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:07:17,302 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 23.15 2021-12-15 14:07:17,318 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:17,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:17,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:25,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.15 % 2021-12-15 14:07:25,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:25,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:27,335 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.03 % 2021-12-15 14:07:27,335 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.03 2021-12-15 14:07:27,351 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:27,354 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:27,354 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:35,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.44 % 2021-12-15 14:07:35,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:35,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:37,368 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.44 % 2021-12-15 14:07:37,368 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.44 2021-12-15 14:07:37,384 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:37,387 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:37,387 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:45,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.44 % 2021-12-15 14:07:45,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:45,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:47,398 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.44 % 2021-12-15 14:07:47,398 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.44 2021-12-15 14:07:47,415 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:47,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:47,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:55,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:07:55,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:07:55,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:07:57,434 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:07:57,434 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.81 2021-12-15 14:07:57,453 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:07:57,456 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:07:57,457 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:05,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:05,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:05,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:07,469 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:07,470 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.81 2021-12-15 14:08:07,486 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:07,489 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:07,490 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:15,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:15,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:15,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:17,503 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:17,503 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.81 2021-12-15 14:08:17,523 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:17,525 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:17,525 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:25,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:25,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:25,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:27,539 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:27,539 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.81 2021-12-15 14:08:27,556 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:27,559 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:27,559 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:35,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:35,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:35,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:37,572 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.81 % 2021-12-15 14:08:37,572 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 24.81 2021-12-15 14:08:37,586 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:37,588 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:37,588 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:45,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:08:45,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:45,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:47,598 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:08:47,598 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 25.27 2021-12-15 14:08:47,614 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:47,615 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:47,615 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:55,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:08:55,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:08:55,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:08:57,628 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:08:57,630 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 25.27 2021-12-15 14:08:57,645 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:08:57,648 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:08:57,648 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:05,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:09:05,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:05,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:07,664 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.27 % 2021-12-15 14:09:07,664 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 25.27 2021-12-15 14:09:07,684 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:07,686 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:07,686 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:15,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.69 % 2021-12-15 14:09:15,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:15,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:17,697 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.06 % 2021-12-15 14:09:17,698 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.06 2021-12-15 14:09:17,717 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:17,720 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:17,720 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:25,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:25,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:25,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:27,734 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:27,736 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.52 2021-12-15 14:09:27,756 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:27,759 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:27,759 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:35,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:35,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:35,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:37,772 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:37,772 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.52 2021-12-15 14:09:37,792 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:37,795 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:37,795 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:45,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:45,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:45,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:47,809 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:47,809 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.52 2021-12-15 14:09:47,823 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:47,826 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:47,826 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:55,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:55,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:09:55,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:09:57,839 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.52 % 2021-12-15 14:09:57,839 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.52 2021-12-15 14:09:57,859 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:09:57,861 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:09:57,862 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:05,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:05,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:05,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:07,877 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:07,877 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.93 2021-12-15 14:10:07,894 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:07,897 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:07,898 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:15,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:15,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:15,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:17,911 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:17,911 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.93 2021-12-15 14:10:17,934 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:17,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:17,937 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:25,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:25,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:25,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:27,953 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:27,953 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 26.93 2021-12-15 14:10:27,971 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:27,974 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:27,974 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:35,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-15 14:10:35,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:35,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:37,989 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.29 % 2021-12-15 14:10:37,989 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 27.29 2021-12-15 14:10:38,006 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:38,009 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:38,009 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:45,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.76 % 2021-12-15 14:10:45,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:45,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:48,036 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.76 % 2021-12-15 14:10:48,036 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 27.76 2021-12-15 14:10:48,055 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:48,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:48,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:55,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:10:55,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:10:55,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:10:58,071 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:10:58,072 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.18 2021-12-15 14:10:58,086 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:10:58,089 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:10:58,089 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:05,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:05,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:05,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:08,104 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:08,104 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.18 2021-12-15 14:11:08,119 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:08,122 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:08,122 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:15,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:15,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:15,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:18,134 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:18,134 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.18 2021-12-15 14:11:18,148 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:18,154 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:18,154 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:25,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:25,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:25,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:28,167 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:28,168 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.18 2021-12-15 14:11:28,188 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:28,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:28,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:36,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:36,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:36,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:38,233 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.18 % 2021-12-15 14:11:38,233 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.18 2021-12-15 14:11:38,250 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:38,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:38,255 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:46,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:11:46,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:46,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:48,266 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:11:48,266 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.64 2021-12-15 14:11:48,282 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:48,285 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:48,285 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:56,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:11:56,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:11:56,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:11:58,295 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:11:58,296 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.64 2021-12-15 14:11:58,311 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:11:58,313 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:11:58,313 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:06,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:12:06,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:06,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:08,325 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.64 % 2021-12-15 14:12:08,325 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 28.64 2021-12-15 14:12:08,362 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:08,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:08,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:16,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.42 % 2021-12-15 14:12:16,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:16,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:18,381 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.42 % 2021-12-15 14:12:18,382 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.42 2021-12-15 14:12:18,401 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:18,405 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:18,405 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:26,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.42 % 2021-12-15 14:12:26,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:26,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:28,420 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:28,420 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:12:28,438 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:28,441 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:28,441 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:36,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:36,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:36,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:38,453 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:38,453 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:12:38,467 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:38,470 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:38,470 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:46,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:46,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:46,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:48,484 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:48,486 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:12:48,500 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:48,503 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:48,503 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:56,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:56,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:12:56,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:12:58,513 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:12:58,513 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:12:58,529 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:12:58,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:12:58,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:06,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:13:06,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:06,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:08,543 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:13:08,545 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:13:08,559 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:08,561 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:08,562 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:16,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:13:16,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:16,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:18,575 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.88 % 2021-12-15 14:13:18,575 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 29.88 2021-12-15 14:13:18,594 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:18,595 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:18,596 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:26,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.30 % 2021-12-15 14:13:26,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:26,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:28,609 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.30 % 2021-12-15 14:13:28,609 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 30.3 2021-12-15 14:13:28,631 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:28,632 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:28,632 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:36,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.71 % 2021-12-15 14:13:36,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:36,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:38,647 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.71 % 2021-12-15 14:13:38,647 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 30.71 2021-12-15 14:13:38,664 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:38,667 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:38,667 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:46,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-15 14:13:46,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:46,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:48,678 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-15 14:13:48,680 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.18 2021-12-15 14:13:48,693 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:48,694 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:48,694 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:56,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:13:56,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:13:56,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:13:58,704 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:13:58,706 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.54 2021-12-15 14:13:58,720 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:13:58,723 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:13:58,723 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:06,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:06,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:06,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:08,733 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:08,734 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.54 2021-12-15 14:14:08,749 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:08,750 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:08,752 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:16,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:16,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:16,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:18,765 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:18,765 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.54 2021-12-15 14:14:18,786 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:18,789 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:18,789 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:26,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:26,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:26,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:28,801 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:28,801 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.54 2021-12-15 14:14:28,816 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:28,819 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:28,819 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:36,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:36,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:36,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:38,832 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.54 % 2021-12-15 14:14:38,832 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.54 2021-12-15 14:14:38,849 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:38,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:38,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:46,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-15 14:14:46,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:46,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:48,862 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-15 14:14:48,862 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.96 2021-12-15 14:14:48,878 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:48,880 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:48,880 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:56,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-15 14:14:56,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:14:56,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:14:58,890 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-15 14:14:58,891 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 31.96 2021-12-15 14:14:58,904 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:14:58,907 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:14:58,907 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:06,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-15 14:15:06,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:06,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:08,918 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.42 % 2021-12-15 14:15:08,920 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 32.42 2021-12-15 14:15:08,938 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:08,941 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:08,941 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:16,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.42 % 2021-12-15 14:15:16,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:16,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:18,953 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-15 14:15:18,953 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 32.79 2021-12-15 14:15:18,970 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:18,973 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:18,973 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:26,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-15 14:15:26,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:26,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:28,984 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:28,984 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 14:15:29,007 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:29,010 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:29,010 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:36,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:36,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:36,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:39,023 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:39,025 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 14:15:39,042 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:39,046 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:39,046 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:46,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:46,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:46,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:49,055 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:49,056 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 14:15:49,072 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:49,075 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:49,075 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:56,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 14:15:56,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:15:56,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:15:59,086 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:15:59,086 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.66 2021-12-15 14:15:59,104 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:15:59,107 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:15:59,107 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:06,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:06,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:06,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:09,118 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:09,118 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.66 2021-12-15 14:16:09,135 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:09,138 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:09,138 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:16,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:16,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:16,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:19,150 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:19,150 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.66 2021-12-15 14:16:19,167 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:19,171 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:19,171 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:26,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:26,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:26,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:29,181 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:29,183 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.66 2021-12-15 14:16:29,197 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:29,200 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:29,200 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:36,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:36,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:36,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:39,216 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.66 % 2021-12-15 14:16:39,216 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 33.66 2021-12-15 14:16:39,233 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:39,236 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:39,236 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:46,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 14:16:46,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:46,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:49,247 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 14:16:49,247 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 14:16:49,263 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:49,266 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:49,266 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:56,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:16:56,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:16:56,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:16:59,282 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:16:59,282 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 34.91 2021-12-15 14:16:59,299 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:16:59,302 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:16:59,302 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:06,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:17:06,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:06,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:09,312 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:17:09,313 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 34.91 2021-12-15 14:17:09,332 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:09,335 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:09,335 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:16,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:17:16,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:16,724 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:19,345 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 14:17:19,345 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 34.91 2021-12-15 14:17:19,361 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:19,365 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:19,365 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:26,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:26,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:26,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:29,375 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:29,375 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.32 2021-12-15 14:17:29,390 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:29,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:29,392 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:36,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:36,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:36,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:39,404 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:39,404 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.32 2021-12-15 14:17:39,421 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:39,424 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:39,424 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:46,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:46,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:46,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:49,433 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:49,434 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.32 2021-12-15 14:17:49,447 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:49,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:49,450 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:56,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:56,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:17:56,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:17:59,461 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:17:59,461 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.32 2021-12-15 14:17:59,477 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:17:59,480 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:17:59,480 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:06,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:18:06,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:06,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:09,494 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.32 % 2021-12-15 14:18:09,494 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.32 2021-12-15 14:18:09,513 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:09,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:09,517 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:16,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.68 % 2021-12-15 14:18:16,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:16,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:19,528 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.68 % 2021-12-15 14:18:19,528 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 35.68 2021-12-15 14:18:19,543 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:19,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:19,546 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:26,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.68 % 2021-12-15 14:18:26,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:26,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:29,559 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.15 % 2021-12-15 14:18:29,559 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 36.15 2021-12-15 14:18:29,576 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:29,579 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:29,581 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:36,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.15 % 2021-12-15 14:18:36,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:36,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:39,589 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.57 % 2021-12-15 14:18:39,591 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 36.57 2021-12-15 14:18:39,605 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:39,608 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:39,608 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:46,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:18:46,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:46,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:49,621 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:18:49,622 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.03 2021-12-15 14:18:49,638 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:49,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:49,641 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:56,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:18:56,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:18:56,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:18:59,653 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:18:59,653 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.03 2021-12-15 14:18:59,670 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:18:59,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:18:59,671 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:06,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:06,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:06,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:09,684 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:09,684 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.03 2021-12-15 14:19:09,700 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:09,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:09,701 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:16,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:16,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:16,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:19,713 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:19,714 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.03 2021-12-15 14:19:19,732 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:19,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:19,733 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:26,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:26,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:26,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:29,760 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.03 % 2021-12-15 14:19:29,762 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.03 2021-12-15 14:19:29,779 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:29,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:29,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:36,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:36,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:36,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:39,795 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:39,796 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.4 2021-12-15 14:19:39,815 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:39,816 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:39,816 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:47,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:47,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:47,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:49,828 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:49,828 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.4 2021-12-15 14:19:49,848 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:49,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:49,851 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:57,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:57,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:19:57,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:19:59,861 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:19:59,861 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.4 2021-12-15 14:19:59,881 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:19:59,884 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:19:59,884 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:07,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.40 % 2021-12-15 14:20:07,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:07,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:09,897 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 14:20:09,898 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 37.81 2021-12-15 14:20:09,914 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:20:09,915 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:20:09,917 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:17,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.81 % 2021-12-15 14:20:17,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:17,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:19,927 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.27 % 2021-12-15 14:20:19,927 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.27 2021-12-15 14:20:19,943 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:20:19,944 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:20:19,944 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:27,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:27,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:27,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:29,954 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:29,956 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.69 2021-12-15 14:20:29,970 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:20:29,971 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:20:29,973 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:37,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:37,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:37,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:39,983 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:39,984 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.69 2021-12-15 14:20:40,009 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:20:40,013 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:20:40,013 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:47,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:47,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:47,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:50,023 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:50,025 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.69 2021-12-15 14:20:50,039 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:20:50,042 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:20:50,042 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:20:57,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:20:57,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:20:57,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:00,052 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:21:00,052 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.69 2021-12-15 14:21:00,068 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:00,069 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:00,069 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:07,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:21:07,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:07,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:10,085 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.69 % 2021-12-15 14:21:10,085 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 38.69 2021-12-15 14:21:10,107 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:10,109 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:10,109 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:17,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:17,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:17,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:20,121 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:20,121 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 39.06 2021-12-15 14:21:20,141 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:20,144 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:20,144 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:27,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:27,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:27,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:30,155 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:30,155 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 39.06 2021-12-15 14:21:30,171 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:30,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:30,174 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:37,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:37,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:37,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:40,184 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.06 % 2021-12-15 14:21:40,184 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 39.06 2021-12-15 14:21:40,200 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:40,201 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:40,203 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:47,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-15 14:21:47,239 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:47,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:50,211 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-15 14:21:50,211 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 39.52 2021-12-15 14:21:50,226 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:21:50,229 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:21:50,229 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:21:57,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-15 14:21:57,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:21:57,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:00,239 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-15 14:22:00,240 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 39.99 2021-12-15 14:22:00,255 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:00,256 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:00,257 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:07,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:07,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:07,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:10,267 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:10,267 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.4 2021-12-15 14:22:10,282 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:10,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:10,285 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:17,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:17,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:17,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:20,293 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:20,295 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.4 2021-12-15 14:22:20,309 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:20,312 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:20,312 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:27,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:27,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:27,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:30,322 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.40 % 2021-12-15 14:22:30,322 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.4 2021-12-15 14:22:30,335 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:30,338 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:30,338 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:37,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:22:37,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:37,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:40,348 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:22:40,348 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.76 2021-12-15 14:22:40,365 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:40,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:40,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:47,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:22:47,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:47,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:50,380 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:22:50,380 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.76 2021-12-15 14:22:50,394 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:22:50,397 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:22:50,397 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:22:57,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:22:57,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:22:57,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:00,407 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:23:00,407 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.76 2021-12-15 14:23:00,423 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:00,424 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:00,424 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:07,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:23:07,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:07,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:10,437 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:23:10,437 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.76 2021-12-15 14:23:10,451 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:10,454 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:10,454 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:17,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:23:17,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:17,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:20,463 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.76 % 2021-12-15 14:23:20,464 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 40.76 2021-12-15 14:23:20,480 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:20,483 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:20,483 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:27,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.18 % 2021-12-15 14:23:27,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:27,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:30,510 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.18 % 2021-12-15 14:23:30,510 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 41.18 2021-12-15 14:23:30,528 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:30,530 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:30,530 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:37,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.18 % 2021-12-15 14:23:37,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:37,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:40,543 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.64 % 2021-12-15 14:23:40,543 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 41.64 2021-12-15 14:23:40,561 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:40,563 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:40,565 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:47,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.11 % 2021-12-15 14:23:47,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:47,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:50,573 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.11 % 2021-12-15 14:23:50,573 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.11 2021-12-15 14:23:50,588 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:23:50,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:23:50,592 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:23:57,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.11 % 2021-12-15 14:23:57,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:23:57,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:00,604 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.11 % 2021-12-15 14:24:00,605 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.11 2021-12-15 14:24:00,625 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:00,628 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:00,628 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:07,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:07,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:07,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:10,641 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:10,641 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.47 2021-12-15 14:24:10,657 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:10,658 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:10,660 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:17,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:17,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:17,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:20,671 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:20,671 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.47 2021-12-15 14:24:20,687 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:20,688 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:20,688 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:27,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:27,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:27,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:30,701 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:30,701 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.47 2021-12-15 14:24:30,717 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:30,719 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:30,720 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:37,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.47 % 2021-12-15 14:24:37,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:37,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:40,730 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.84 % 2021-12-15 14:24:40,730 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.84 2021-12-15 14:24:40,744 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:40,746 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:40,747 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:47,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.84 % 2021-12-15 14:24:47,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:47,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:50,759 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.84 % 2021-12-15 14:24:50,759 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.84 2021-12-15 14:24:50,780 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:24:50,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:24:50,783 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:24:57,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.84 % 2021-12-15 14:24:57,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:24:57,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:00,793 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.84 % 2021-12-15 14:25:00,793 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 42.84 2021-12-15 14:25:00,812 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:00,813 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:00,813 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:07,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:07,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:07,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:10,825 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:10,825 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 43.26 2021-12-15 14:25:10,839 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:10,842 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:10,842 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:17,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:17,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:17,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:20,852 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:20,852 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 43.26 2021-12-15 14:25:20,875 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:20,877 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:20,877 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:27,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:27,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:27,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:30,888 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.26 % 2021-12-15 14:25:30,888 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 43.26 2021-12-15 14:25:30,907 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:30,910 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:30,910 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:37,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.72 % 2021-12-15 14:25:37,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:37,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:40,921 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.19 % 2021-12-15 14:25:40,921 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.19 2021-12-15 14:25:40,946 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:40,947 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:40,947 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:47,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.19 % 2021-12-15 14:25:47,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:47,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:50,959 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.19 % 2021-12-15 14:25:50,959 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.19 2021-12-15 14:25:50,973 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:25:50,976 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:25:50,976 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:25:57,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.19 % 2021-12-15 14:25:57,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:25:57,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:00,986 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:00,987 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.55 2021-12-15 14:26:01,009 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:01,012 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:01,012 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:07,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:07,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:07,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:11,023 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:11,025 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.55 2021-12-15 14:26:11,046 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:11,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:11,048 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:17,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:17,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:17,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:21,058 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:21,059 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.55 2021-12-15 14:26:21,073 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:21,076 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:21,076 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:27,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:27,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:27,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:31,088 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.55 % 2021-12-15 14:26:31,089 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.55 2021-12-15 14:26:31,105 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:31,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:31,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:37,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.91 % 2021-12-15 14:26:37,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:37,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:41,118 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.91 % 2021-12-15 14:26:41,118 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 44.91 2021-12-15 14:26:41,137 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:41,138 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:41,140 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:47,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.91 % 2021-12-15 14:26:47,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:47,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:51,150 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.38 % 2021-12-15 14:26:51,150 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 45.38 2021-12-15 14:26:51,164 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:26:51,167 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:26:51,167 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:26:57,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.38 % 2021-12-15 14:26:57,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:26:57,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:01,177 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.38 % 2021-12-15 14:27:01,177 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 45.38 2021-12-15 14:27:01,194 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:01,197 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:01,197 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:07,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.38 % 2021-12-15 14:27:07,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:07,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:11,207 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:11,209 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 45.8 2021-12-15 14:27:11,223 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:11,226 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:11,226 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:17,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:17,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:17,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:21,236 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:21,236 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 45.8 2021-12-15 14:27:21,250 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:21,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:21,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:27,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:27,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:27,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:31,265 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:31,265 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 45.8 2021-12-15 14:27:31,279 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:31,282 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:31,282 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:37,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.80 % 2021-12-15 14:27:37,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:37,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:41,292 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.26 % 2021-12-15 14:27:41,292 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 46.26 2021-12-15 14:27:41,308 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:41,311 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:41,311 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:47,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.26 % 2021-12-15 14:27:47,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:47,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:51,322 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.26 % 2021-12-15 14:27:51,322 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 46.26 2021-12-15 14:27:51,338 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:27:51,339 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:27:51,339 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:27:57,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:27:57,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:27:57,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:01,349 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:28:01,349 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 46.63 2021-12-15 14:28:01,365 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:01,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:01,368 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:07,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:28:08,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:08,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:11,378 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:28:11,378 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 46.63 2021-12-15 14:28:11,394 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:11,397 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:11,397 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:18,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:28:18,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:18,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:21,405 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.63 % 2021-12-15 14:28:21,407 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 46.63 2021-12-15 14:28:21,423 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:21,426 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:21,426 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:28,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:28,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:28,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:31,434 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:31,434 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.04 2021-12-15 14:28:31,463 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:31,466 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:31,466 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:38,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:38,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:38,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:41,479 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:41,480 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.04 2021-12-15 14:28:41,494 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:41,496 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:41,497 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:48,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:48,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:48,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:51,509 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.04 % 2021-12-15 14:28:51,509 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.04 2021-12-15 14:28:51,528 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:28:51,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:28:51,532 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:28:58,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.50 % 2021-12-15 14:28:58,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:28:58,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:01,542 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.50 % 2021-12-15 14:29:01,542 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.5 2021-12-15 14:29:01,559 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:01,562 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:01,562 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:08,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.50 % 2021-12-15 14:29:08,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:08,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:11,573 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.50 % 2021-12-15 14:29:11,573 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.5 2021-12-15 14:29:11,588 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:11,589 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:11,589 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:18,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.87 % 2021-12-15 14:29:18,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:18,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:21,601 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.87 % 2021-12-15 14:29:21,601 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 47.87 2021-12-15 14:29:21,618 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:21,621 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:21,621 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:28,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.87 % 2021-12-15 14:29:28,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:28,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:31,631 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:31,631 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-15 14:29:31,648 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:31,651 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:31,651 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:38,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:38,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:38,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:41,661 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:41,661 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-15 14:29:41,677 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:41,678 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:41,680 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:48,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:48,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:48,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:51,688 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:51,690 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-15 14:29:51,703 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:29:51,706 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:29:51,706 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:29:58,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:29:58,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:29:58,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:01,716 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:30:01,716 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-15 14:30:01,746 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:01,747 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:01,749 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:08,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:30:08,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:08,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:11,762 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:30:11,762 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-15 14:30:11,778 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:11,779 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:11,779 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:18,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-15 14:30:18,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:18,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:21,790 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 14:30:21,790 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.64 2021-12-15 14:30:21,805 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:21,806 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:21,806 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:28,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 14:30:28,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:28,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:31,819 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 14:30:31,819 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 48.64 2021-12-15 14:30:31,835 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:31,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:31,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:38,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.05 % 2021-12-15 14:30:38,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:38,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:41,849 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.05 % 2021-12-15 14:30:41,849 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.05 2021-12-15 14:30:41,869 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:41,871 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:41,871 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:48,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.05 % 2021-12-15 14:30:48,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:48,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:51,881 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.41 % 2021-12-15 14:30:51,882 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.41 2021-12-15 14:30:51,903 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:30:51,904 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:30:51,904 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:30:58,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.41 % 2021-12-15 14:30:58,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:30:58,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:01,914 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:01,914 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.83 2021-12-15 14:31:01,933 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:01,934 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:01,934 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:08,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:08,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:08,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:11,947 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:11,947 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.83 2021-12-15 14:31:11,966 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:11,969 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:11,969 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:18,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:18,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:18,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:21,980 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:21,980 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.83 2021-12-15 14:31:22,000 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:22,003 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:22,003 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:28,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:28,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:28,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:32,013 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.83 % 2021-12-15 14:31:32,015 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 49.83 2021-12-15 14:31:32,029 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:32,030 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:32,030 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:38,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:31:38,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:38,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:42,040 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:31:42,040 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 50.25 2021-12-15 14:31:42,055 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:42,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:42,059 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:48,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:31:48,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:48,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:52,072 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:31:52,072 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 50.25 2021-12-15 14:31:52,088 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:31:52,091 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:31:52,091 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:31:58,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:31:58,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:31:58,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:02,101 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:32:02,101 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 50.25 2021-12-15 14:32:02,117 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:02,119 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:02,119 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:08,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:32:08,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:08,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:12,131 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:32:12,131 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 50.25 2021-12-15 14:32:12,148 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:12,151 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:12,151 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:18,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.25 % 2021-12-15 14:32:18,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:18,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:22,163 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.61 % 2021-12-15 14:32:22,164 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 50.61 2021-12-15 14:32:22,180 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:22,183 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:22,183 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:28,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.08 % 2021-12-15 14:32:28,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:28,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:32,194 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:32,196 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.55 2021-12-15 14:32:32,211 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:32,213 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:32,214 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:38,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:38,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:38,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:42,230 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:42,230 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.55 2021-12-15 14:32:42,250 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:42,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:42,253 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:48,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:48,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:48,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:52,263 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:52,263 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.55 2021-12-15 14:32:52,280 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:32:52,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:32:52,283 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:32:58,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:32:58,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:32:58,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:02,293 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:33:02,293 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.55 2021-12-15 14:33:02,309 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:02,312 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:02,312 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:08,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:33:08,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:08,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:12,322 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:33:12,323 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.55 2021-12-15 14:33:12,342 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:12,345 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:12,345 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:18,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.55 % 2021-12-15 14:33:18,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:18,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:22,357 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:22,357 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.91 2021-12-15 14:33:22,372 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:22,375 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:22,375 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:28,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:28,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:28,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:32,384 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:32,384 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.91 2021-12-15 14:33:32,403 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:32,407 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:32,407 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:38,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:38,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:38,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:42,418 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:42,418 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.91 2021-12-15 14:33:42,437 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:42,440 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:42,441 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:48,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:48,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:48,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:52,456 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:52,456 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 51.91 2021-12-15 14:33:52,473 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:33:52,476 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:33:52,476 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:33:58,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.91 % 2021-12-15 14:33:58,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:33:58,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:02,486 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.33 % 2021-12-15 14:34:02,486 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 52.33 2021-12-15 14:34:02,500 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:02,503 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:02,503 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:08,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.74 % 2021-12-15 14:34:08,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:08,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:12,513 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.74 % 2021-12-15 14:34:12,513 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 52.74 2021-12-15 14:34:12,533 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:12,536 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:12,536 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:18,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:18,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:18,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:22,548 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:22,548 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.2 2021-12-15 14:34:22,563 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:22,566 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:22,566 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:28,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:28,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:28,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:32,581 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:32,581 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.2 2021-12-15 14:34:32,598 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:32,601 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:32,601 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:38,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:38,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:38,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:42,611 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:42,611 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.2 2021-12-15 14:34:42,627 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:42,630 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:42,631 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:48,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:48,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:48,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:52,641 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.20 % 2021-12-15 14:34:52,641 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.2 2021-12-15 14:34:52,657 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:34:52,660 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:34:52,660 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:34:58,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.57 % 2021-12-15 14:34:58,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:34:58,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:02,671 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.57 % 2021-12-15 14:35:02,671 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.57 2021-12-15 14:35:02,687 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:02,688 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:02,688 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:08,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.57 % 2021-12-15 14:35:08,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:08,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:12,698 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.57 % 2021-12-15 14:35:12,698 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.57 2021-12-15 14:35:12,716 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:12,719 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:12,719 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:18,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:18,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:18,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:22,729 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:22,730 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.99 2021-12-15 14:35:22,744 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:22,747 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:22,747 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:28,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:28,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:28,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:32,760 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:32,760 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.99 2021-12-15 14:35:32,779 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:32,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:32,782 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:38,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:38,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:38,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:42,790 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.99 % 2021-12-15 14:35:42,790 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 53.99 2021-12-15 14:35:42,805 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:42,806 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:42,806 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:48,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.45 % 2021-12-15 14:35:48,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:48,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:52,816 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.45 % 2021-12-15 14:35:52,816 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 54.45 2021-12-15 14:35:52,832 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:35:52,834 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:35:52,834 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:35:58,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.45 % 2021-12-15 14:35:58,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:35:58,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:02,844 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.86 % 2021-12-15 14:36:02,844 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 54.86 2021-12-15 14:36:02,858 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:02,859 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:02,859 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:08,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.86 % 2021-12-15 14:36:09,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:09,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:12,871 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.86 % 2021-12-15 14:36:12,871 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 54.86 2021-12-15 14:36:12,888 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:12,890 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:12,891 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:19,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:19,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:19,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:22,901 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:22,901 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.22 2021-12-15 14:36:22,921 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:22,923 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:22,923 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:29,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:29,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:29,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:32,933 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:32,933 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.22 2021-12-15 14:36:32,948 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:32,950 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:32,950 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:39,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:39,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:39,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:42,967 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.22 % 2021-12-15 14:36:42,967 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.22 2021-12-15 14:36:42,990 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:42,993 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:42,993 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:49,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:36:49,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:49,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:53,003 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:36:53,003 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.69 2021-12-15 14:36:53,019 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:36:53,020 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:36:53,020 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:36:59,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:36:59,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:36:59,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:03,030 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:03,032 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.69 2021-12-15 14:37:03,046 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:03,049 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:03,049 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:09,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:09,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:09,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:13,059 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:13,059 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.69 2021-12-15 14:37:13,075 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:13,075 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:13,076 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:19,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:19,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:19,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:23,086 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:23,086 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 55.69 2021-12-15 14:37:23,105 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:23,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:23,108 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:29,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.69 % 2021-12-15 14:37:29,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:29,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:33,118 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.11 % 2021-12-15 14:37:33,118 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 56.11 2021-12-15 14:37:33,134 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:33,135 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:33,137 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:39,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:37:39,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:39,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:43,147 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:37:43,147 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 56.94 2021-12-15 14:37:43,161 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:43,164 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:43,164 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:49,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:37:49,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:49,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:53,177 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:37:53,177 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 56.94 2021-12-15 14:37:53,191 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:37:53,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:37:53,194 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:37:59,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:37:59,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:37:59,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:03,203 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:38:03,203 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 56.94 2021-12-15 14:38:03,217 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:03,219 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:03,219 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:09,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:38:09,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:09,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:13,227 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.94 % 2021-12-15 14:38:13,229 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 56.94 2021-12-15 14:38:13,242 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:13,243 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:13,243 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:19,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:19,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:19,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:23,255 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:23,255 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:38:23,276 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:23,279 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:23,279 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:29,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:29,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:29,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:33,292 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:33,292 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:38:33,321 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:33,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:33,322 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:39,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:39,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:39,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:43,341 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:43,341 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:38:43,355 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:43,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:43,358 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:49,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:49,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:49,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:53,368 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:53,368 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:38:53,384 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:38:53,385 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:38:53,385 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:38:59,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:38:59,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:38:59,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:03,395 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:39:03,395 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:39:03,414 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:03,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:03,418 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:09,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:39:09,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:09,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:13,433 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.35 % 2021-12-15 14:39:13,433 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 57.35 2021-12-15 14:39:13,450 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:13,453 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:13,453 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:19,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.71 % 2021-12-15 14:39:19,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:19,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:23,463 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.18 % 2021-12-15 14:39:23,463 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 58.18 2021-12-15 14:39:23,482 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:23,484 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:23,484 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:29,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.64 % 2021-12-15 14:39:29,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:29,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:33,496 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.64 % 2021-12-15 14:39:33,496 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 58.64 2021-12-15 14:39:33,513 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:33,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:33,516 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:39,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:39:39,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:39,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:43,528 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:39:43,529 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:39:43,546 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:43,549 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:43,549 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:49,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:39:49,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:49,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:53,559 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:39:53,559 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:39:53,578 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:39:53,579 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:39:53,579 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:39:59,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:39:59,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:39:59,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:03,592 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:03,592 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:40:03,608 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:03,611 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:03,611 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:09,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:09,433 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:09,433 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:13,621 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:13,621 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:40:13,635 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:13,638 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:13,638 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:19,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:19,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:19,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:23,648 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:23,648 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:40:23,661 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:23,664 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:23,664 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:29,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:29,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:29,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:33,674 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:33,674 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:40:33,688 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:33,691 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:33,691 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:39,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:39,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:39,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:43,700 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.11 % 2021-12-15 14:40:43,700 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.11 2021-12-15 14:40:43,714 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:43,717 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:43,717 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:49,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.47 % 2021-12-15 14:40:49,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:49,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:53,726 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.47 % 2021-12-15 14:40:53,726 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.47 2021-12-15 14:40:53,742 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:40:53,744 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:40:53,744 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:40:59,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.47 % 2021-12-15 14:40:59,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:40:59,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:41:03,755 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.47 % 2021-12-15 14:41:03,755 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 59.47 2021-12-15 14:41:03,769 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:41:03,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:41:03,772 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:41:09,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.82 % 2021-12-15 14:41:09,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:41:09,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:41:13,782 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.82 % 2021-12-15 14:41:13,783 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 60.82 2021-12-15 14:41:13,802 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:41:13,805 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:41:13,805 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:41:19,601 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >60.82% in 3256.704 seconds 2021-12-15 14:41:19,609 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 14:41:19,609 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 14:41:19,611 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 14:41:19,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:41:19,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 14:41:19,980 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:41:19,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 14:41:19,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 14:41:20,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 14:41:20,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 14:41:20,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 14:41:20,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 14:41:20,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 14:41:20,118 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 14:41:20,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 14:41:20,174 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['114536 memcached\n'], []) 2021-12-15 14:41:20,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 14:41:20,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 14:41:20,263 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['114536 memcached\n'], []) 2021-12-15 14:41:20,263 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 14:41:20,266 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 14:41:20,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:41:20,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 14:41:20,625 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:41:20,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 14:41:20,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 14:41:20,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 14:41:20,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 14:41:20,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 14:41:20,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 14:41:20,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 14:41:20,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:41:20,755 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 14:41:20,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:41:20,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:20,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:20,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:20,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:20,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:21,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:21,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:21,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:21,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:21,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:21,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:21,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:21,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:21,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:21,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 14:41:21,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 14:41:21,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:21,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:21,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 14:41:21,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 14:41:21,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:41:21,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 14:41:21,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:41:21,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:21,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:41:21,591 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 14:41:21,592 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 14:41:21,595 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 14:41:21,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:41:22,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 14:41:22,119 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:41:22,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 14:41:22,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 14:41:22,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 14:41:22,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 14:41:22,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 14:41:22,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 14:41:22,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 14:41:22,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:41:22,283 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 14:41:22,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:41:22,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:22,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:22,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:22,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:22,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:22,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:22,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:22,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:22,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:22,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:22,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:22,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:22,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:41:22,849 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 14:41:22,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:41:22,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:22,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:41:22,911 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 14:41:22,911 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 14:41:22,914 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 14:41:23,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:41:23,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 14:41:23,348 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:41:23,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 14:41:23,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 14:41:23,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 14:41:23,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 14:41:23,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 14:41:23,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 14:41:23,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 14:41:23,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:41:23,453 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 14:41:23,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:41:23,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:23,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:23,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:23,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:23,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:41:23,609 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 14:41:23,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:41:23,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:23,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:41:23,647 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 14:41:23,647 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 14:41:23,650 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 14:41:23,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:41:23,816 | test | DEBUG | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.82 % 2021-12-15 14:41:23,816 | test | INFO | pool-3-thread-22 | [task:check:385] Rebalance - status: running, progress: 60.82 2021-12-15 14:41:23,836 | infra | DEBUG | pool-3-thread-22 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:41:23,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:41:23,838 | infra | DEBUG | pool-3-thread-22 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:41:24,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 14:41:24,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:41:24,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 14:41:24,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 14:41:24,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 14:41:24,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 14:41:24,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 14:41:24,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 14:41:24,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 14:41:24,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:41:24,369 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 14:41:24,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:41:24,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:24,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:24,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:24,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:41:24,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:24,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:24,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:24,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:41:24,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:24,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:41:24,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:24,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:41:24,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:41:24,778 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 14:41:24,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:41:24,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:41:24,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:41:24,818 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 14:41:24,818 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 14:41:24,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 14:41:24,855 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 14:41:33,848 | test | ERROR | pool-3-thread-22 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'af15e5af8cce3ee8513909b665e11a83', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b2801928f322a805fb81ac51efb0f81f', u'status': u'notRunning'} - rebalance failed 2021-12-15 14:41:33,865 | test | INFO | pool-3-thread-22 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 14:41:33,865 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092228L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.228Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.19866.122>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,506}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,264,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.5199.117>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 264}},\n infinity]}}}}}.\nRebalance Operation Id = eaf0694a1d7bc371fbe5e373173ebc54'} 2021-12-15 14:41:33,865 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092205L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.205Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 13:39:48.673293 114770 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 14:41:33,865 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092187L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.187Z', u'text': u'Worker <0.19662.122> (for action {move,\n {264,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.19866.122>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n stats_recv,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 163}]},\n {mc_client_binary,\n stats,4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 455}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 506}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 225}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 264,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.5199.117>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 264}},\n infinity]}}}}'} 2021-12-15 14:41:33,867 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092173L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.173Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 14:41:33,867 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604703284L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:45:03.284Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 14:41:33,867 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604703120L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:45:03.120Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 14:41:33,867 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604702903L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:45:02.903Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = eaf0694a1d7bc371fbe5e373173ebc54"} 2021-12-15 14:41:33,867 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604689675L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:44:49.675Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 273 seconds.'} 2021-12-15 14:41:33,868 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384068L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.068Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.17549.116>,\n {{{{nocatch,{error,closed}},\n [{mc_binary,recv_with_data,4,\n [{file,"src/mc_binary.erl"},{line,41}]},\n {mc_binary,quick_active_recv,3,\n [{file,"src/mc_binary.erl"},{line,48}]},\n {mc_binary,quick_stats_loop_enter,5,\n [{file,"src/mc_binary.erl"},{line,100}]},\n {mc_binary,quick_stats,5,\n [{file,"src/mc_binary.erl"},{line,85}]},\n {mc_client_binary,get_dcp_docs_estimate,\n 3,\n [{file,"src/mc_client_binary.erl"},\n {line,754}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,616}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,121,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.18323.110>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 291}},\n infinity]}}}}}.\nRebalance Operation Id = d19fc0aba12ea674050f5f5a0a03f06f'} 2021-12-15 14:41:33,868 | test | ERROR | pool-3-thread-22 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639604384059L, u'shortText': u'message', u'serverTime': u'2021-12-15T13:39:44.059Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 12:31:49.944420 109296 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 14:41:33,868 | test | ERROR | pool-3-thread-22 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'af15e5af8cce3ee8513909b665e11a83', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b2801928f322a805fb81ac51efb0f81f', u'status': u'notRunning'} - rebalance failed 2021-12-15 14:42:24,868 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 14:42:24,869 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 14:42:24,881 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:24,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:24,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:26,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:26,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:26,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:28,923 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:28,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:28,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:30,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:30,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:30,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:32,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:32,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:32,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:34,986 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:34,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:34,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:37,005 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:37,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:37,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:39,028 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:39,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:39,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:41,051 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:41,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:41,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:43,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:43,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:43,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:45,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:45,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:45,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:47,109 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:47,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:47,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:49,130 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:49,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:49,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:51,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:51,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:51,155 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:53,167 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:53,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:53,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:55,186 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:55,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:55,191 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:57,203 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:57,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:57,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:42:59,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:42:59,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:42:59,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:01,243 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:01,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:01,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:03,262 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:03,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:03,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:05,286 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:05,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:05,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:07,308 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:07,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:07,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:09,331 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:09,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:09,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:11,351 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:11,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:11,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:13,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:13,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:13,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:15,417 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:15,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:15,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:17,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:17,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:17,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:19,454 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:19,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:19,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:21,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:21,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:21,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:23,497 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:23,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:23,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:25,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:25,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:25,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:27,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:27,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:27,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:29,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:29,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:29,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:31,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:31,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:31,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:33,596 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:33,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:33,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:35,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:35,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:35,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:37,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:37,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:37,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:39,670 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:39,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:39,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:41,688 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:41,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:41,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:43,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:43,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:43,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:45,732 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:45,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:45,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:47,750 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:47,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:47,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:49,770 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:49,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:49,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:51,793 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:51,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:51,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:53,815 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:53,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:53,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:55,835 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:55,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:55,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:57,855 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:57,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:57,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:43:59,878 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:43:59,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:43:59,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:01,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:01,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:01,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:03,920 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:03,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:03,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:05,971 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:05,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:05,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:07,993 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:08,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:08,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:10,016 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:10,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:10,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:12,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:12,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:12,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:14,063 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:14,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:14,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:16,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:16,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:16,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:18,115 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:18,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:18,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:20,135 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:20,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:20,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:22,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:22,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:22,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:24,178 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:24,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:24,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:26,197 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:26,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:26,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:28,216 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:28,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:28,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:30,236 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:30,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:30,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:32,255 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:32,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:32,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:34,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:34,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:34,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:36,293 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:36,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:36,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:38,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:38,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:38,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:40,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:40,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:40,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:42,358 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:42,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:42,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:44,378 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:44,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:44,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:46,400 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:46,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:46,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:48,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:48,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:48,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:50,443 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:50,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:50,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:52,463 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:52,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:52,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:54,480 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:54,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:54,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:56,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:56,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:56,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:44:58,523 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:44:58,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:44:58,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:00,545 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:00,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:00,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:02,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:02,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:02,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:04,584 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:04,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:04,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:06,602 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:06,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:06,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:08,625 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:08,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:08,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:10,647 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:10,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:10,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:12,668 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:12,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:12,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:14,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:14,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:14,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:16,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:16,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:16,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:18,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:18,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:18,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:20,763 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:20,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:20,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:22,811 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:22,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:22,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:24,836 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:24,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:24,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:26,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:26,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:26,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:28,878 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:28,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:28,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:30,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:30,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:30,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:32,920 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:32,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:32,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:34,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:34,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:34,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:36,964 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:36,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:36,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:38,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:38,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:38,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:41,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:41,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:41,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:43,036 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:43,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:43,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:45,058 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 14:45:45,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 14:45:45,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 14:45:47,081 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 14:45:47,098 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 14:45:47,099 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 14:45:47,115 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 14:45:47,128 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-15 14:45:47,128 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 14:45:47,140 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-15 14:45:47,154 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 14:45:47,154 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 14:45:47,167 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 14:45:47,170 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 14:45:47,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:45:47,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 14:45:47,585 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:45:47,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 14:45:47,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 14:45:47,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 14:45:47,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 14:45:47,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 14:45:47,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 14:45:47,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 14:45:47,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:45:47,694 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 14:45:47,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:45:47,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:47,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:47,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:47,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:47,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:47,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:47,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:47,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:47,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:47,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:48,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:48,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:48,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:48,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:48,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 14:45:48,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 14:45:48,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:48,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:48,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 14:45:48,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 14:45:48,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:48,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:48,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 14:45:48,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 14:45:48,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:45:48,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 14:45:48,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:45:48,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:48,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:45:48,446 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 14:45:48,446 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 14:45:48,450 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 14:45:48,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:45:48,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 14:45:48,874 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:45:48,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 14:45:48,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 14:45:48,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 14:45:48,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 14:45:48,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 14:45:48,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 14:45:48,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 14:45:48,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:45:48,974 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 14:45:49,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:45:49,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:49,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:49,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:49,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:49,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:49,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:49,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:49,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:49,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:49,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:49,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:49,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:49,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:45:49,450 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 14:45:49,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:45:49,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:49,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:45:49,484 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 14:45:49,484 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 14:45:49,487 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 14:45:49,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:45:49,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 14:45:49,865 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:45:49,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 14:45:49,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 14:45:49,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 14:45:49,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 14:45:49,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 14:45:49,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 14:45:49,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 14:45:49,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:45:49,982 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 14:45:50,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:45:50,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:50,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:50,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:50,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:50,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:45:50,121 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 14:45:50,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:45:50,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:50,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:45:50,163 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 14:45:50,163 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 14:45:50,168 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 14:45:50,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 14:45:50,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 14:45:50,588 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 14:45:50,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 14:45:50,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 14:45:50,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 14:45:50,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 14:45:50,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 14:45:50,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 14:45:50,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 14:45:50,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 14:45:50,687 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 14:45:50,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 14:45:50,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:50,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:50,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:50,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 14:45:50,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:50,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:50,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:50,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 14:45:50,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:50,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 14:45:50,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:50,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 14:45:51,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 14:45:51,012 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 14:45:51,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 14:45:51,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 14:45:51,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 14:45:51,045 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 14:45:51,045 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 14:45:51,046 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1639604702.84 2021-12-15 14:45:51,046 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 60 2021-12-15 14:45:51,078 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1639608351.05 2021-12-15 14:45:51,078 | infra | INFO | pool-3-thread-23 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[]_1639608351.05' started 2021-12-15 14:45:51,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 14:45:51,085 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 14:45:51,111 | test | DEBUG | pool-3-thread-23 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-15 14:45:51,168 | test | DEBUG | pool-3-thread-23 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 14:45:51,171 | test | INFO | pool-3-thread-23 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 10.5038975501 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 1.52659019812 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 0.549977084288 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 0.828249419809 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-15 14:45:51,181 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:45:51,181 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:45:51,197 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:45:51,198 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:45:51,200 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:01,213 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:01,213 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:01,234 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:01,236 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:01,237 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:11,265 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:11,266 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:11,279 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:11,282 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:11,282 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:21,292 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:21,292 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:21,309 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:21,312 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:21,312 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:31,323 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:31,323 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:31,341 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:31,344 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:31,344 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:41,357 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:41,357 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:41,374 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:41,377 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:41,377 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:46:51,388 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:46:51,388 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:46:51,411 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:46:51,414 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:46:51,414 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:01,424 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 14:47:01,424 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 14:47:01,440 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:01,444 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:01,444 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:11,453 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.70 % 2021-12-15 14:47:11,453 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.7 2021-12-15 14:47:11,470 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:11,471 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:11,473 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:21,483 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.70 % 2021-12-15 14:47:21,483 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 0.7 2021-12-15 14:47:21,497 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:21,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:21,500 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:31,510 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.50 % 2021-12-15 14:47:31,512 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 2.5 2021-12-15 14:47:31,529 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:31,532 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:31,532 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:41,542 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.50 % 2021-12-15 14:47:41,542 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 2.5 2021-12-15 14:47:41,556 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:41,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:41,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:51,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:47:51,095 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 5.67 2021-12-15 14:47:51,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:47:51,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:47:51,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:47:51,569 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:47:51,569 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 14:47:51,585 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:47:51,586 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:47:51,586 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:01,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:01,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:01,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:01,596 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:01,596 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 14:48:01,612 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:01,615 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:01,615 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:11,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:11,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:11,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:11,627 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:11,627 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 14:48:11,641 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:11,644 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:11,644 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:21,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:21,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:21,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:21,655 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:21,655 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 14:48:21,671 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:21,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:21,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:31,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:31,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:31,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:31,684 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.67 % 2021-12-15 14:48:31,684 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 5.67 2021-12-15 14:48:31,700 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:31,703 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:31,703 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:41,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.37 % 2021-12-15 14:48:41,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:41,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:41,713 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.37 % 2021-12-15 14:48:41,713 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 6.37 2021-12-15 14:48:41,727 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:41,730 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:41,730 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:51,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.37 % 2021-12-15 14:48:51,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:48:51,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:48:51,740 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.37 % 2021-12-15 14:48:51,740 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 6.37 2021-12-15 14:48:51,755 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:48:51,757 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:48:51,757 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:01,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-15 14:49:01,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:01,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:01,770 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-15 14:49:01,770 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-15 14:49:01,789 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:01,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:01,792 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:11,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.87 % 2021-12-15 14:49:11,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:11,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:11,803 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.87 % 2021-12-15 14:49:11,805 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 8.87 2021-12-15 14:49:11,822 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:11,826 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:11,826 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:21,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.87 % 2021-12-15 14:49:21,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:21,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:21,838 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.87 % 2021-12-15 14:49:21,838 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 8.87 2021-12-15 14:49:21,855 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:21,858 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:21,858 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:31,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:31,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:31,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:31,868 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:31,868 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-15 14:49:31,888 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:31,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:31,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:41,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:41,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:41,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:41,904 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:41,904 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-15 14:49:41,921 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:41,923 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:41,924 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:51,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:51,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:49:51,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:49:51,933 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:49:51,933 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-15 14:49:51,947 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:49:51,948 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:49:51,948 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:01,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:50:01,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:01,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:01,957 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:50:01,959 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-15 14:50:01,974 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:01,977 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:01,977 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:11,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:50:11,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:11,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:11,989 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-15 14:50:11,989 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-15 14:50:12,005 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:12,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:12,006 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:21,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.37 % 2021-12-15 14:50:21,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:21,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:22,016 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.37 % 2021-12-15 14:50:22,016 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 11.37 2021-12-15 14:50:22,032 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:22,033 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:22,033 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:31,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.37 % 2021-12-15 14:50:31,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:31,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:32,042 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.37 % 2021-12-15 14:50:32,043 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 11.37 2021-12-15 14:50:32,056 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:32,059 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:32,059 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:41,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 14:50:41,433 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:41,433 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:42,068 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 14:50:42,068 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 12.51 2021-12-15 14:50:42,082 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:42,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:42,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:51,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 14:50:51,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:50:51,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:50:52,095 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 14:50:52,095 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 12.51 2021-12-15 14:50:52,114 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:50:52,115 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:50:52,115 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:01,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:01,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:01,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:02,127 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:02,127 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 14:51:02,141 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:02,144 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:02,144 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:11,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:11,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:11,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:12,154 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:12,154 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 14:51:12,171 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:12,173 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:12,173 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:21,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:21,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:21,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:22,184 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:22,186 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 14:51:22,204 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:22,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:22,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:31,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:31,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:31,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:32,217 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 14:51:32,219 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 14:51:32,240 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:32,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:32,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:41,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:51:41,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:41,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:42,253 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:51:42,253 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 16.37 2021-12-15 14:51:42,270 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:42,273 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:42,273 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:51,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:51:51,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:51:51,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:51:52,285 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:51:52,286 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 16.37 2021-12-15 14:51:52,305 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:51:52,306 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:51:52,306 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:01,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:52:01,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:01,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:02,318 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 14:52:02,318 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 16.37 2021-12-15 14:52:02,341 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:02,342 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:02,342 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:11,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-15 14:52:11,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:11,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:12,355 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-15 14:52:12,355 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 18.18 2021-12-15 14:52:12,375 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:12,377 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:12,377 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:21,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-15 14:52:21,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:21,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:22,388 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-15 14:52:22,388 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 18.18 2021-12-15 14:52:22,410 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:22,413 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:22,413 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:31,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 14:52:31,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:31,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:32,423 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 14:52:32,423 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 14:52:32,437 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:32,440 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:32,440 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:41,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 14:52:41,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:41,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:42,453 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 14:52:42,453 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 14:52:42,471 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:42,476 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:42,476 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:51,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 14:52:51,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:52:51,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:52:52,490 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.13 % 2021-12-15 14:52:52,490 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 21.13 2021-12-15 14:52:52,506 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:52:52,509 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:52:52,509 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:01,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:01,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:01,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:02,523 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:02,525 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 21.82 2021-12-15 14:53:02,539 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:02,542 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:02,542 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:11,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:11,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:11,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:12,559 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:12,559 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 21.82 2021-12-15 14:53:12,578 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:12,582 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:12,582 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:21,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:21,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:21,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:22,598 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.82 % 2021-12-15 14:53:22,598 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 21.82 2021-12-15 14:53:22,617 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:22,621 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:22,621 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:31,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:31,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:31,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:32,632 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:32,634 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 23.19 2021-12-15 14:53:32,648 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:32,651 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:32,651 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:41,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:41,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:41,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:42,661 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:42,661 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 23.19 2021-12-15 14:53:42,680 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:42,683 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:42,683 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:51,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:51,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:53:51,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:53:52,693 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:53:52,693 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 23.19 2021-12-15 14:53:52,707 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:53:52,709 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:53:52,709 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:01,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:54:01,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:01,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:02,719 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.19 % 2021-12-15 14:54:02,719 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 23.19 2021-12-15 14:54:02,736 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:02,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:02,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:11,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.99 % 2021-12-15 14:54:11,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:11,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:12,747 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.99 % 2021-12-15 14:54:12,747 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 24.99 2021-12-15 14:54:12,762 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:12,763 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:12,763 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:21,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.99 % 2021-12-15 14:54:21,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:21,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:22,773 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.99 % 2021-12-15 14:54:22,773 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 24.99 2021-12-15 14:54:22,792 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:22,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:22,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:31,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.13 % 2021-12-15 14:54:31,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:31,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:32,803 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.13 % 2021-12-15 14:54:32,803 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 26.13 2021-12-15 14:54:32,822 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:32,823 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:32,825 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:41,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.83 % 2021-12-15 14:54:41,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:41,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:42,835 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.83 % 2021-12-15 14:54:42,835 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 26.83 2021-12-15 14:54:42,855 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:42,858 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:42,858 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:51,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:54:51,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:54:51,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:54:52,867 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:54:52,868 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 28.63 2021-12-15 14:54:52,884 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:54:52,887 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:54:52,887 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:01,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:01,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:01,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:02,897 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:02,897 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 28.63 2021-12-15 14:55:02,914 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:02,917 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:02,917 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:11,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:11,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:11,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:12,946 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:12,946 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 28.63 2021-12-15 14:55:12,963 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:12,966 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:12,967 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:21,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:21,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:21,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:22,977 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.63 % 2021-12-15 14:55:22,977 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 28.63 2021-12-15 14:55:22,994 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:22,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:22,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:31,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:31,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:31,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:33,007 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:33,007 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 30.0 2021-12-15 14:55:33,022 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:33,025 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:33,025 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:41,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:42,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:42,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:43,036 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:43,036 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 30.0 2021-12-15 14:55:43,052 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:43,055 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:43,055 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:52,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:52,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:55:52,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:55:53,065 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:55:53,065 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 30.0 2021-12-15 14:55:53,082 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:55:53,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:55:53,085 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:02,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:56:02,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:02,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:03,094 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.00 % 2021-12-15 14:56:03,095 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 30.0 2021-12-15 14:56:03,109 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:03,112 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:03,112 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:12,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.69 % 2021-12-15 14:56:12,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:12,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:13,124 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.69 % 2021-12-15 14:56:13,124 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 30.69 2021-12-15 14:56:13,138 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:13,140 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:13,140 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:22,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.50 % 2021-12-15 14:56:22,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:22,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:23,148 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.50 % 2021-12-15 14:56:23,148 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 32.5 2021-12-15 14:56:23,163 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:23,164 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:23,164 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:32,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.50 % 2021-12-15 14:56:32,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:32,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:33,174 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.50 % 2021-12-15 14:56:33,174 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 32.5 2021-12-15 14:56:33,190 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:33,193 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:33,193 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:42,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.64 % 2021-12-15 14:56:42,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:42,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:43,203 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.64 % 2021-12-15 14:56:43,203 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 33.64 2021-12-15 14:56:43,217 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:43,219 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:43,219 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:52,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.64 % 2021-12-15 14:56:52,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:56:52,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:56:53,227 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.64 % 2021-12-15 14:56:53,227 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 33.64 2021-12-15 14:56:53,240 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:56:53,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:56:53,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:02,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:02,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:02,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:03,253 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:03,253 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 35.44 2021-12-15 14:57:03,269 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:03,272 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:03,272 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:12,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:12,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:12,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:13,280 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:13,282 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 35.44 2021-12-15 14:57:13,296 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:13,299 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:13,299 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:22,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:22,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:22,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:23,309 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:23,309 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 35.44 2021-12-15 14:57:23,323 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:23,325 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:23,325 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:32,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:32,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:32,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:33,335 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.44 % 2021-12-15 14:57:33,336 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 35.44 2021-12-15 14:57:33,355 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:33,358 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:33,358 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:42,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.81 % 2021-12-15 14:57:42,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:42,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:43,368 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.81 % 2021-12-15 14:57:43,368 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 36.81 2021-12-15 14:57:43,382 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:43,384 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:43,384 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:52,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.50 % 2021-12-15 14:57:52,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:57:52,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:57:53,395 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.50 % 2021-12-15 14:57:53,395 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 37.5 2021-12-15 14:57:53,414 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:57:53,417 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:57:53,417 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:02,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.50 % 2021-12-15 14:58:02,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:02,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:03,427 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:03,427 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:03,446 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:03,448 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:03,448 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:12,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:12,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:12,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:13,459 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:13,460 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:13,483 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:13,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:13,487 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:22,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:22,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:22,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:23,497 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:23,497 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:23,513 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:23,516 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:23,516 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:32,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:32,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:32,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:33,525 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:33,526 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:33,539 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:33,542 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:33,542 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:42,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:42,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:42,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:43,553 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:43,555 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:43,573 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:43,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:43,576 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:52,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:52,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:58:52,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:58:53,586 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.31 % 2021-12-15 14:58:53,586 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 39.31 2021-12-15 14:58:53,605 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:58:53,608 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:58:53,608 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:02,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.45 % 2021-12-15 14:59:02,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:02,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:03,621 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.45 % 2021-12-15 14:59:03,621 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 40.45 2021-12-15 14:59:03,637 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:03,640 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:03,640 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:12,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.14 % 2021-12-15 14:59:12,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:12,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:13,651 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-15 14:59:13,653 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 42.95 2021-12-15 14:59:13,671 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:13,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:13,674 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:22,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-15 14:59:22,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:22,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:23,686 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-15 14:59:23,686 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 42.95 2021-12-15 14:59:23,703 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:23,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:23,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:32,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-15 14:59:32,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:32,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:33,717 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-15 14:59:33,717 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 42.95 2021-12-15 14:59:33,734 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:33,737 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:33,737 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:42,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 14:59:42,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:42,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:43,753 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 14:59:43,753 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 44.31 2021-12-15 14:59:43,767 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:43,770 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:43,770 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:52,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 14:59:52,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 14:59:52,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 14:59:53,782 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 14:59:53,782 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 44.31 2021-12-15 14:59:53,796 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 14:59:53,799 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 14:59:53,799 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:02,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:02,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:02,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:03,809 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:03,809 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 44.31 2021-12-15 15:00:03,825 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:03,828 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:03,828 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:12,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:12,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:12,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:13,838 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:13,839 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 44.31 2021-12-15 15:00:13,858 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:13,861 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:13,861 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:22,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:22,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:22,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:23,871 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.31 % 2021-12-15 15:00:23,871 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 44.31 2021-12-15 15:00:23,887 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:23,890 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:23,890 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:32,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.12 % 2021-12-15 15:00:32,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:32,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:33,898 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.82 % 2021-12-15 15:00:33,900 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 46.82 2021-12-15 15:00:33,914 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:33,917 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:33,917 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:42,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.82 % 2021-12-15 15:00:42,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:42,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:43,928 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.82 % 2021-12-15 15:00:43,928 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 46.82 2021-12-15 15:00:43,944 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:43,947 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:43,947 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:52,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.82 % 2021-12-15 15:00:52,568 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:00:52,568 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:00:53,957 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.82 % 2021-12-15 15:00:53,957 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 46.82 2021-12-15 15:00:53,973 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:00:53,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:00:53,976 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:02,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:02,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:02,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:03,986 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:03,986 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:04,003 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:04,005 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:04,005 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:12,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:12,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:12,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:14,016 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:14,016 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:14,032 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:14,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:14,036 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:22,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:22,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:22,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:24,046 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:24,046 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:24,061 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:24,063 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:24,063 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:32,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:32,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:32,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:34,073 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:34,073 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:34,088 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:34,091 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:34,092 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:42,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:42,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:42,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:44,102 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:44,102 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:44,121 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:44,125 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:44,125 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:52,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:52,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:01:52,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:01:54,135 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:01:54,135 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:01:54,151 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:01:54,154 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:01:54,154 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:02,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:02:02,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:02,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:04,164 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.76 % 2021-12-15 15:02:04,164 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 49.76 2021-12-15 15:02:04,180 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:04,181 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:04,181 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:12,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.82 % 2021-12-15 15:02:12,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:12,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:14,191 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.82 % 2021-12-15 15:02:14,191 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 51.82 2021-12-15 15:02:14,206 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:14,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:14,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:22,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.96 % 2021-12-15 15:02:22,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:22,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:24,217 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.96 % 2021-12-15 15:02:24,217 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 52.96 2021-12-15 15:02:24,232 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:24,233 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:24,234 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:32,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.96 % 2021-12-15 15:02:32,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:32,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:34,244 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.96 % 2021-12-15 15:02:34,244 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 52.96 2021-12-15 15:02:34,263 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:34,266 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:34,266 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:42,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:02:42,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:42,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:44,275 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:02:44,275 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 54.77 2021-12-15 15:02:44,288 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:44,289 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:44,289 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:52,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:02:52,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:02:52,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:02:54,302 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:02:54,302 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 54.77 2021-12-15 15:02:54,322 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:02:54,323 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:02:54,323 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:02,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:02,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:02,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:04,332 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:04,334 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 54.77 2021-12-15 15:03:04,355 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:04,357 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:04,357 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:12,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:12,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:12,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:14,368 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:14,369 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 54.77 2021-12-15 15:03:14,391 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:14,391 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:14,392 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:22,857 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:22,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:22,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:24,403 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.77 % 2021-12-15 15:03:24,403 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 54.77 2021-12-15 15:03:24,424 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:24,426 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:24,427 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:32,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:32,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:32,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:34,436 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:34,436 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 55.46 2021-12-15 15:03:34,467 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:34,470 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:34,470 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:42,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:42,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:42,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:44,480 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:44,480 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 55.46 2021-12-15 15:03:44,496 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:44,499 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:44,499 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:52,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:52,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:03:52,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:03:54,509 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.46 % 2021-12-15 15:03:54,509 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 55.46 2021-12-15 15:03:54,528 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:03:54,530 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:03:54,530 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:02,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.96 % 2021-12-15 15:04:02,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:02,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:04,539 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.96 % 2021-12-15 15:04:04,540 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 57.96 2021-12-15 15:04:04,556 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:04,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:04,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:12,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.96 % 2021-12-15 15:04:12,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:12,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:14,569 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.96 % 2021-12-15 15:04:14,571 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 57.96 2021-12-15 15:04:14,588 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:14,589 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:14,589 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:22,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.96 % 2021-12-15 15:04:22,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:22,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:24,599 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:24,599 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 59.77 2021-12-15 15:04:24,619 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:24,622 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:24,622 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:32,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:33,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:33,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:34,632 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:34,632 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 59.77 2021-12-15 15:04:34,653 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:34,655 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:34,655 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:43,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:43,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:43,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:44,667 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:44,667 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 59.77 2021-12-15 15:04:44,684 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:44,687 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:44,687 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:53,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.77 % 2021-12-15 15:04:53,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:04:53,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:04:54,698 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:04:54,698 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 60.47 2021-12-15 15:04:54,716 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:04:54,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:04:54,717 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:03,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:03,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:03,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:04,730 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:04,730 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 60.47 2021-12-15 15:05:04,755 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:04,759 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:04,759 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:13,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:13,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:13,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:14,773 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:14,773 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 60.47 2021-12-15 15:05:14,792 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:14,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:14,795 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:23,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:23,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:23,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:24,805 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:24,805 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 60.47 2021-12-15 15:05:24,822 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:24,825 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:24,825 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:33,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:33,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:33,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:34,835 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.47 % 2021-12-15 15:05:34,836 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 60.47 2021-12-15 15:05:34,851 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:34,855 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:34,855 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:43,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.61 % 2021-12-15 15:05:43,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:43,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:44,864 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.61 % 2021-12-15 15:05:44,865 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 61.61 2021-12-15 15:05:44,878 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:44,881 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:44,881 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:53,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.30 % 2021-12-15 15:05:53,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:05:53,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:05:54,891 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.30 % 2021-12-15 15:05:54,891 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 62.3 2021-12-15 15:05:54,907 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:05:54,910 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:05:54,911 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:03,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.30 % 2021-12-15 15:06:03,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:03,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:04,921 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.30 % 2021-12-15 15:06:04,921 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 62.3 2021-12-15 15:06:04,937 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:04,940 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:04,940 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:13,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:13,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:13,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:14,951 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:14,951 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 63.44 2021-12-15 15:06:14,969 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:14,971 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:14,971 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:23,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:23,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:23,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:24,986 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:24,986 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 63.44 2021-12-15 15:06:25,009 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:25,013 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:25,013 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:33,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:33,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:33,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:35,042 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.44 % 2021-12-15 15:06:35,042 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 63.44 2021-12-15 15:06:35,061 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:35,062 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:35,063 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:43,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:06:43,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:43,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:45,075 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:06:45,075 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 64.14 2021-12-15 15:06:45,092 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:45,095 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:45,095 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:53,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:06:53,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:06:53,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:06:55,104 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:06:55,105 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 64.14 2021-12-15 15:06:55,119 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:06:55,122 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:06:55,122 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:03,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:07:03,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:03,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:05,135 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:07:05,137 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 64.14 2021-12-15 15:07:05,151 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:05,154 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:05,154 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:13,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:07:13,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:13,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:15,164 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:07:15,164 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 64.14 2021-12-15 15:07:15,178 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:15,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:15,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:23,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.14 % 2021-12-15 15:07:23,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:23,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:25,190 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.28 % 2021-12-15 15:07:25,190 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 65.28 2021-12-15 15:07:25,204 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:25,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:25,207 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:33,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:33,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:33,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:35,217 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:35,217 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 65.97 2021-12-15 15:07:35,234 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:35,236 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:35,236 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:43,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:43,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:43,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:45,246 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:45,246 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 65.97 2021-12-15 15:07:45,260 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:45,263 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:45,263 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:53,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:53,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:07:53,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:07:55,273 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.97 % 2021-12-15 15:07:55,273 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 65.97 2021-12-15 15:07:55,286 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:07:55,288 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:07:55,288 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:03,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.11 % 2021-12-15 15:08:03,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:03,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:05,298 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.11 % 2021-12-15 15:08:05,298 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.11 2021-12-15 15:08:05,312 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:05,313 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:05,313 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:13,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.11 % 2021-12-15 15:08:13,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:13,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:15,325 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.11 % 2021-12-15 15:08:15,325 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.11 2021-12-15 15:08:15,338 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:15,341 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:15,341 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:23,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:23,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:23,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:25,351 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:25,352 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.81 2021-12-15 15:08:25,372 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:25,375 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:25,375 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:33,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:33,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:33,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:35,384 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:35,385 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.81 2021-12-15 15:08:35,400 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:35,401 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:35,401 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:43,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:43,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:43,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:45,413 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:45,413 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.81 2021-12-15 15:08:45,430 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:45,431 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:45,431 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:53,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:53,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:08:53,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:08:55,441 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:08:55,441 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.81 2021-12-15 15:08:55,457 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:08:55,459 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:08:55,459 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:03,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:09:03,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:03,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:05,470 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.81 % 2021-12-15 15:09:05,470 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 67.81 2021-12-15 15:09:05,493 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:05,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:05,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:13,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.50 % 2021-12-15 15:09:13,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:13,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:15,509 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.64 % 2021-12-15 15:09:15,509 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 69.64 2021-12-15 15:09:15,526 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:15,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:15,529 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:23,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.64 % 2021-12-15 15:09:23,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:23,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:25,542 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.64 % 2021-12-15 15:09:25,542 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 69.64 2021-12-15 15:09:25,562 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:25,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:25,565 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:33,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.64 % 2021-12-15 15:09:33,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:33,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:35,575 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:09:35,575 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:09:35,589 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:35,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:35,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:43,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:09:43,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:43,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:45,602 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:09:45,602 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:09:45,618 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:45,619 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:45,619 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:53,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:09:53,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:09:53,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:09:55,630 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:09:55,631 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:09:55,648 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:09:55,650 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:09:55,651 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:03,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:03,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:03,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:05,663 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:05,663 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:10:05,677 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:05,680 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:05,680 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:13,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:13,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:13,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:15,690 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:15,690 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:10:15,706 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:15,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:15,707 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:23,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:23,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:23,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:25,719 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:25,719 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:10:25,736 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:25,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:25,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:33,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:33,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:33,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:35,750 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:35,750 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:10:35,773 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:35,775 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:35,776 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:43,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:43,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:43,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:45,785 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.78 % 2021-12-15 15:10:45,785 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 70.78 2021-12-15 15:10:45,799 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:45,802 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:45,802 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:53,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:10:53,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:10:53,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:10:55,812 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:10:55,812 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:10:55,828 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:10:55,832 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:10:55,832 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:03,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:03,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:03,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:05,842 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:05,842 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:05,858 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:05,861 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:05,861 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:13,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:13,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:13,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:15,871 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:15,871 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:15,890 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:15,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:15,891 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:23,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:23,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:23,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:25,907 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:25,907 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:25,924 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:25,926 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:25,926 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:33,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:33,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:33,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:35,936 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:35,936 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:35,953 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:35,956 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:35,956 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:43,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:43,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:43,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:45,967 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:45,967 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:45,982 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:45,983 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:45,984 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:53,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:53,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:11:53,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:11:55,996 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:11:55,996 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:11:56,013 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:11:56,016 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:11:56,016 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:03,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:03,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:03,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:06,025 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:06,025 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:12:06,040 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:06,042 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:06,042 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:13,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:13,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:13,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:16,052 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:16,052 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:12:16,066 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:16,069 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:16,069 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:23,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:23,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:23,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:26,078 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.48 % 2021-12-15 15:12:26,079 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 71.48 2021-12-15 15:12:26,095 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:26,098 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:26,098 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:33,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.17 % 2021-12-15 15:12:33,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:33,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:36,108 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.17 % 2021-12-15 15:12:36,109 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 72.17 2021-12-15 15:12:36,125 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:36,128 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:36,128 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:43,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 15:12:43,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:43,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:46,138 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 15:12:46,138 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 72.87 2021-12-15 15:12:46,154 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:46,157 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:46,157 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:53,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 15:12:53,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:12:53,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:12:56,165 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 15:12:56,167 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 72.87 2021-12-15 15:12:56,180 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:12:56,184 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:12:56,184 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:03,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 15:13:03,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:03,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:06,194 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:06,194 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:06,209 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:06,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:06,211 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:13,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:13,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:13,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:16,221 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:16,221 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:16,236 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:16,239 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:16,239 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:23,984 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:23,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:23,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:26,249 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:26,249 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:26,265 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:26,267 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:26,267 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:34,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:34,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:34,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:36,278 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:36,279 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:36,295 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:36,298 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:36,298 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:44,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:44,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:44,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:46,309 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:46,309 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:46,325 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:46,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:46,328 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:54,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:54,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:13:54,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:13:56,336 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:13:56,336 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:13:56,351 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:13:56,354 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:13:56,354 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:04,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:14:04,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:04,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:06,362 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.57 % 2021-12-15 15:14:06,364 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 73.57 2021-12-15 15:14:06,377 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:06,378 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:06,378 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:14,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:14,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:14,096 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:16,388 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:16,390 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:14:16,404 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:16,405 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:16,407 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:24,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:24,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:24,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:26,417 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:26,417 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:14:26,436 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:26,437 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:26,437 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:34,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:34,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:34,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:36,448 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:36,448 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:14:36,464 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:36,467 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:36,467 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:44,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:44,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:44,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:46,477 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:46,477 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:14:46,494 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:46,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:46,496 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:54,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:54,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:14:54,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:14:56,506 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:14:56,506 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:14:56,528 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:14:56,530 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:14:56,530 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:04,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:04,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:04,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:06,542 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:06,542 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:15:06,555 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:06,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:06,559 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:14,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:14,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:14,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:16,569 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:16,571 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:15:16,591 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:16,592 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:16,594 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:24,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:24,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:24,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:26,605 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:26,605 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:15:26,624 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:26,625 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:26,625 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:34,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:34,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:34,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:36,635 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:36,635 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:15:36,674 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:36,677 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:36,677 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:44,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:44,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:44,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:46,687 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.26 % 2021-12-15 15:15:46,688 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.26 2021-12-15 15:15:46,707 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:46,710 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:46,710 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:54,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.96 % 2021-12-15 15:15:54,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:15:54,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:15:56,719 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.96 % 2021-12-15 15:15:56,720 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 74.96 2021-12-15 15:15:56,736 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:15:56,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:15:56,739 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:04,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.65 % 2021-12-15 15:16:04,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:04,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:06,749 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.65 % 2021-12-15 15:16:06,750 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 75.65 2021-12-15 15:16:06,769 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:06,773 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:06,773 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:14,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.65 % 2021-12-15 15:16:14,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:14,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:16,785 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.65 % 2021-12-15 15:16:16,785 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 75.65 2021-12-15 15:16:16,805 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:16,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:16,809 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:24,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.65 % 2021-12-15 15:16:24,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:24,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:26,818 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:26,818 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:16:26,834 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:26,835 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:26,836 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:34,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:34,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:34,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:36,846 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:36,846 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:16:36,861 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:36,865 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:36,865 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:44,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:44,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:44,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:46,880 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:46,880 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:16:46,894 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:46,898 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:46,898 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:54,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:54,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:16:54,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:16:56,908 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:16:56,910 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:16:56,927 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:16:56,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:16:56,930 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:04,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:04,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:04,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:06,941 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:06,941 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:17:06,961 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:06,964 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:06,964 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:14,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:14,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:14,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:16,976 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:16,976 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:17:16,993 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:16,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:16,997 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:24,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:24,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:24,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:27,012 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-15 15:17:27,012 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 76.35 2021-12-15 15:17:27,029 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:27,032 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:27,032 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:34,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:34,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:34,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:37,046 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:37,046 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:17:37,062 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:37,065 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:37,065 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:44,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:44,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:44,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:47,075 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:47,075 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:17:47,088 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:47,092 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:47,092 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:54,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:54,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:17:54,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:17:57,101 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:17:57,102 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:17:57,118 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:17:57,121 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:17:57,121 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:04,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:04,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:04,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:07,131 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:07,131 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:07,148 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:07,151 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:07,151 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:14,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:14,568 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:14,568 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:17,161 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:17,161 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:17,177 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:17,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:17,180 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:24,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:24,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:24,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:27,194 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:27,194 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:27,210 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:27,213 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:27,213 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:34,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:34,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:34,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:37,223 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:37,224 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:37,240 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:37,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:37,243 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:44,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:44,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:44,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:47,253 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:47,255 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:47,269 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:47,273 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:47,273 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:54,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:54,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:18:54,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:18:57,283 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:18:57,285 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:18:57,302 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:18:57,305 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:18:57,306 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:04,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:19:04,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:04,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:07,316 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:19:07,318 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:19:07,335 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:07,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:07,338 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:14,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:19:14,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:14,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:17,351 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.04 % 2021-12-15 15:19:17,351 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.04 2021-12-15 15:19:17,368 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:17,371 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:17,371 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:24,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.74 % 2021-12-15 15:19:24,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:24,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:27,381 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.74 % 2021-12-15 15:19:27,381 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 77.74 2021-12-15 15:19:27,403 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:27,404 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:27,405 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:34,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:34,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:34,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:37,417 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:37,417 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 78.43 2021-12-15 15:19:37,431 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:37,434 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:37,434 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:44,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:44,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:44,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:47,444 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:47,444 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 78.43 2021-12-15 15:19:47,459 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:47,460 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:47,460 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:54,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:54,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:19:54,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:19:57,470 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.43 % 2021-12-15 15:19:57,471 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 78.43 2021-12-15 15:19:57,493 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:19:57,494 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:19:57,494 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:04,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:04,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:04,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:07,503 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:07,505 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.13 2021-12-15 15:20:07,523 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:07,526 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:07,526 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:14,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:14,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:14,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:17,536 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:17,536 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.13 2021-12-15 15:20:17,553 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:17,556 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:17,556 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:24,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:24,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:24,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:27,571 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:27,571 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.13 2021-12-15 15:20:27,586 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:27,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:27,588 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:34,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:34,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:34,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:37,598 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:37,598 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.13 2021-12-15 15:20:37,614 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:37,617 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:37,617 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:44,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:44,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:44,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:47,628 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.13 % 2021-12-15 15:20:47,628 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.13 2021-12-15 15:20:47,648 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:47,650 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:47,650 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:54,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:20:54,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:20:54,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:20:57,660 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:20:57,660 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:20:57,674 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:20:57,676 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:20:57,676 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:04,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:04,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:04,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:07,686 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:07,686 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:07,698 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:07,700 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:07,700 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:14,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:14,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:14,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:17,713 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:17,713 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:17,733 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:17,734 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:17,734 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:24,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:24,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:24,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:27,746 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:27,746 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:27,762 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:27,765 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:27,765 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:34,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:34,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:34,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:37,776 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:37,776 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:37,798 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:37,801 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:37,801 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:44,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:44,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:44,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:47,812 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:47,812 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:47,825 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:47,828 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:47,829 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:54,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:54,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:21:54,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:21:57,838 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:21:57,838 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:21:57,858 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:21:57,859 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:21:57,859 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:05,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:05,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:05,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:07,871 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:07,871 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:22:07,885 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:07,887 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:07,888 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:15,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:15,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:15,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:17,897 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:17,898 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:22:17,913 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:17,915 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:17,915 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:25,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:25,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:25,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:27,930 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:27,930 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:22:27,951 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:27,954 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:27,954 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:35,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:35,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:35,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:37,964 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:37,966 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 79.83 2021-12-15 15:22:37,983 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:37,986 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:37,986 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:45,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.83 % 2021-12-15 15:22:45,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:45,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:47,996 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.52 % 2021-12-15 15:22:47,996 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 80.52 2021-12-15 15:22:48,029 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:48,032 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:48,032 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:55,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.22 % 2021-12-15 15:22:55,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 15:22:55,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:22:58,042 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.22 % 2021-12-15 15:22:58,043 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 81.22 2021-12-15 15:22:58,059 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:22:58,062 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:22:58,062 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:23:05,124 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >81.22% in 2114.02900004 seconds 2021-12-15 15:23:05,132 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 15:23:05,134 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 15:23:05,135 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:23:05,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:23:05,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:23:05,469 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:23:05,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:23:05,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:23:05,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:23:05,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:23:05,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:23:05,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:23:05,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:23:05,562 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:23:05,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:23:05,651 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['119681 memcached\n'], []) 2021-12-15 15:23:05,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 15:23:05,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:23:05,740 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['119681 memcached\n'], []) 2021-12-15 15:23:05,740 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:23:05,743 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:23:05,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:23:06,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:23:06,174 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:23:06,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:23:06,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:23:06,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:23:06,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:23:06,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:23:06,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:23:06,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:23:06,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:23:06,323 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 15:23:06,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:23:06,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:06,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:06,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:06,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:06,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:06,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:06,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:06,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:06,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:06,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:06,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:06,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:06,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:06,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:06,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:23:06,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:23:06,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:06,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:06,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:23:07,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:23:07,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:07,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:07,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:23:07,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:23:07,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:23:07,121 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 15:23:07,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:23:07,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:07,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:23:07,163 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 15:23:07,163 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:23:07,165 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:23:07,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:23:07,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:23:07,663 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:23:07,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:23:07,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:23:07,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:23:07,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:23:07,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:23:07,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:23:07,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:23:07,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:23:07,823 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 15:23:07,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:23:07,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:07,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:07,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:08,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:08,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:08,072 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.22 % 2021-12-15 15:23:08,072 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 81.22 2021-12-15 15:23:08,094 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:23:08,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:08,095 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:23:08,095 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:23:08,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:08,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:08,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:08,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:08,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:08,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:08,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:23:08,368 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 15:23:08,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:23:08,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:08,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:23:08,423 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 15:23:08,423 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:23:08,426 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:23:08,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:23:08,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:23:08,857 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:23:08,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:23:08,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:23:08,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:23:08,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:23:08,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:23:08,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:23:08,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:23:08,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:23:09,006 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 15:23:09,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:23:09,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:09,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:09,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:09,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:09,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:23:09,181 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 15:23:09,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:23:09,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:09,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:23:09,243 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 15:23:09,243 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:23:09,246 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:23:09,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:23:09,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:23:09,759 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:23:09,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:23:09,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:23:09,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:23:09,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:23:09,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:23:09,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:23:09,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:23:09,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:23:09,933 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 15:23:09,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:23:09,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:10,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:10,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:10,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:23:10,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:10,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:10,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:10,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:23:10,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:10,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:23:10,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:10,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:23:10,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:23:10,450 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 15:23:10,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:23:10,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:23:10,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:23:10,503 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 15:23:10,503 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:23:10,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 15:23:10,509 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 15:23:18,105 | test | DEBUG | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.22 % 2021-12-15 15:23:18,107 | test | INFO | pool-3-thread-23 | [task:check:385] Rebalance - status: running, progress: 81.22 2021-12-15 15:23:18,125 | infra | DEBUG | pool-3-thread-23 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:23:18,128 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:23:18,128 | infra | DEBUG | pool-3-thread-23 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:23:28,138 | test | ERROR | pool-3-thread-23 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c2e9d6657c35f0f1b90f3c19abd5b9b4', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=dee86c7b768d2ae1989bb4d72b348a80', u'status': u'notRunning'} - rebalance failed 2021-12-15 15:23:28,157 | test | INFO | pool-3-thread-23 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639610598666L, u'shortText': u'message', u'serverTime': u'2021-12-15T15:23:18.666Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.11811.126>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",794,\'ns_1@172.23.110.66\',\n [\'ns_1@172.23.110.68\',\'ns_1@172.23.110.64\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,451}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 24c5b72e8dea3d7898928f8167b2d2e5'} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639610598636L, u'shortText': u'message', u'serverTime': u'2021-12-15T15:23:18.636Z', u'text': u'Worker <0.12192.126> (for action {move,\n {794,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.64\'],\n [\'ns_1@172.23.110.68\',\n \'ns_1@172.23.110.64\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.11811.126>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",\n 794,\n \'ns_1@172.23.110.66\',\n [\'ns_1@172.23.110.68\',\n \'ns_1@172.23.110.64\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 451}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639610596650L, u'shortText': u'message', u'serverTime': u'2021-12-15T15:23:16.650Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 14:41:40.787086 119914 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639610596613L, u'shortText': u'message', u'serverTime': u'2021-12-15T15:23:16.613Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608351514L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:45:51.514Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608351356L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:45:51.356Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608351114L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:45:51.114Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.68',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 24c5b72e8dea3d7898928f8167b2d2e5"} 2021-12-15 15:23:28,157 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608344863L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:45:44.863Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 225 seconds.'} 2021-12-15 15:23:28,158 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092228L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.228Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.19866.122>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,stats_recv,4,\n [{file,"src/mc_client_binary.erl"},\n {line,163}]},\n {mc_client_binary,stats,4,\n [{file,"src/mc_client_binary.erl"},\n {line,455}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,506}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,264,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.5199.117>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\',\n \'ns_1@172.23.110.68\'],\n 264}},\n infinity]}}}}}.\nRebalance Operation Id = eaf0694a1d7bc371fbe5e373173ebc54'} 2021-12-15 15:23:28,158 | test | ERROR | pool-3-thread-23 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639608092205L, u'shortText': u'message', u'serverTime': u'2021-12-15T14:41:32.205Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 13:39:48.673293 114770 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 15:23:28,158 | test | ERROR | pool-3-thread-23 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c2e9d6657c35f0f1b90f3c19abd5b9b4', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=dee86c7b768d2ae1989bb4d72b348a80', u'status': u'notRunning'} - rebalance failed 2021-12-15 15:24:10,522 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 15:24:10,523 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 15:24:10,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:10,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:10,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:12,549 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:12,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:12,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:14,568 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:14,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:14,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:16,585 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:16,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:16,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:18,604 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:18,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:18,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:20,624 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:20,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:20,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:22,642 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:22,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:22,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:24,660 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:24,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:24,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:26,677 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:26,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:26,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:28,694 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:28,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:28,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:30,711 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:30,719 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:30,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:32,732 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:32,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:32,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:34,750 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:34,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:34,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:36,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:36,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:36,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:38,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:38,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:38,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:40,812 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:40,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:40,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:42,829 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:42,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:42,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:44,849 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:44,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:44,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:46,868 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:46,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:46,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:48,888 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:48,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:48,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:50,907 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:50,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:50,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:52,924 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:52,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:52,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:54,944 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:54,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:54,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:56,963 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:56,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:56,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:24:58,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:24:58,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:24:58,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:01,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:01,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:01,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:03,020 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:03,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:03,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:05,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:05,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:05,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:07,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:07,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:07,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:09,081 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:09,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:09,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:11,102 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:11,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:11,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:13,124 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:13,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:13,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:15,141 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:15,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:15,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:17,161 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:17,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:17,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:19,181 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:19,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:19,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:21,203 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:21,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:21,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:23,221 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:23,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:23,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:25,242 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:25,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:25,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:27,259 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:27,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:27,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:29,278 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:29,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:29,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:31,296 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:31,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:31,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:33,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:33,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:33,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:35,335 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:35,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:35,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:37,355 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:37,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:37,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:39,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:39,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:39,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:41,401 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:41,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:41,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:43,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:43,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:43,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:45,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:45,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:45,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:47,460 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:47,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:47,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:49,483 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:49,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:49,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:51,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:51,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:51,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:53,539 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:53,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:53,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:55,561 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:55,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:55,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:57,581 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:57,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:57,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:25:59,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:25:59,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:25:59,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:01,622 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:01,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:01,630 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:03,642 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:03,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:03,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:05,663 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:05,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:05,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:07,683 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:07,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:07,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:09,703 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:09,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:09,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:11,723 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:11,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:11,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:13,743 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:13,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:13,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:15,766 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:15,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:15,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:17,789 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:17,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:17,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:19,812 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:19,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:19,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:21,835 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:21,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:21,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:23,855 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:23,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:23,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:25,877 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:25,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:25,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:27,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:27,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:27,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:29,917 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:29,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:29,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:31,934 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:31,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:31,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:33,953 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:33,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:33,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:35,976 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:35,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:35,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:37,997 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:38,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:38,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:40,015 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:40,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:40,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:42,033 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:42,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:42,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:44,053 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:44,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:44,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:46,072 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:46,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:46,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:48,096 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:48,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:48,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:50,122 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:50,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:50,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:52,140 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:52,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:52,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:54,158 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:54,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:54,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:56,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:56,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:56,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:26:58,206 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:26:58,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:26:58,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:27:00,226 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:27:00,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:27:00,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:27:02,246 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 15:27:02,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 15:27:02,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 15:27:04,269 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 15:27:04,279 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 15:27:04,279 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 15:27:04,289 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 15:27:04,299 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-15 15:27:04,299 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 15:27:04,309 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-15 15:27:04,319 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 15:27:04,321 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 15:27:04,331 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 15:27:04,332 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:27:04,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:27:04,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:27:04,743 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:27:04,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:27:04,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:27:04,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:27:04,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:27:04,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:27:04,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:27:04,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:27:04,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:27:04,890 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 15:27:04,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:27:04,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:04,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:05,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:05,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:05,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:05,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:05,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:05,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:27:05,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:27:05,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:27:05,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:27:05,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:05,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:27:05,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:27:05,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:27:05,490 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 15:27:05,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:27:05,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:05,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:27:05,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 15:27:05,539 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:27:05,542 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:27:05,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:27:05,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:27:05,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:27:05,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:27:05,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:27:06,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:27:06,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:27:06,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:27:06,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:27:06,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:27:06,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:27:06,095 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 15:27:06,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:27:06,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:06,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:06,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:06,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:06,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:06,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:06,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:06,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:06,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:06,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:06,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:06,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:06,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:27:06,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 15:27:06,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:27:06,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:06,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:27:06,436 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 15:27:06,436 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:27:06,438 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:27:06,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:27:06,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:27:06,859 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:27:06,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:27:06,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:27:06,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:27:06,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:27:06,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:27:06,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:27:06,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:27:07,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:27:07,026 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 15:27:07,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:27:07,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:07,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:07,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:07,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:07,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:27:07,160 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 15:27:07,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:27:07,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:07,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:27:07,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 15:27:07,197 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:27:07,201 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:27:07,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:27:07,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:27:07,581 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:27:07,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:27:07,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:27:07,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:27:07,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:27:07,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:27:07,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:27:07,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:27:07,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:27:07,680 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 15:27:07,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:27:07,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:07,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:07,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:07,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:27:07,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:07,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:07,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:07,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:27:08,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:08,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:27:08,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:08,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:27:08,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:27:08,107 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 15:27:08,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:27:08,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:27:08,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:27:08,145 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 15:27:08,145 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:27:08,145 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1639608351.05 2021-12-15 15:27:08,145 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 80 2021-12-15 15:27:08,201 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1639610828.15 2021-12-15 15:27:08,203 | infra | INFO | pool-3-thread-24 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[]_1639610828.15' started 2021-12-15 15:27:08,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 15:27:08,216 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 15:27:08,226 | test | DEBUG | pool-3-thread-24 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-15 15:27:08,279 | test | DEBUG | pool-3-thread-24 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 15:27:08,282 | test | INFO | pool-3-thread-24 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 10.5828284938 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 4.28074084389 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 1.14559003448 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 2.70164992627 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 15:27:08,290 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:08,292 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:08,306 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:08,309 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:08,309 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:27:18,322 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:18,322 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:18,336 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:18,338 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:18,338 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:27:28,351 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:28,351 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:28,365 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:28,368 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:28,368 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:27:38,381 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:38,381 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:38,395 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:38,397 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:38,398 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:27:48,411 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:48,411 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:48,433 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:48,434 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:48,434 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:27:58,448 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:27:58,448 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:27:58,470 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:27:58,471 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:27:58,471 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:08,484 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:08,484 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:08,499 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:08,500 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:08,500 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:18,515 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:18,515 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:18,532 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:18,535 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:18,536 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:28,549 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:28,549 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:28,562 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:28,565 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:28,565 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:38,578 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:38,578 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:38,594 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:38,595 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:38,596 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:48,618 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:48,618 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:48,644 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:48,647 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:48,647 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:28:58,660 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:28:58,661 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:28:58,681 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:28:58,683 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:28:58,684 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:08,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:08,230 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.0 2021-12-15 15:29:08,230 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1639610828.15 2021-12-15 15:29:08,694 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:08,694 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:08,716 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:08,717 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:08,717 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:18,732 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:18,732 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:18,753 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:18,755 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:18,755 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:28,772 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:28,772 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:28,805 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:28,808 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:28,808 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:38,822 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:38,822 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:38,844 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:38,845 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:38,845 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:48,858 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:48,858 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:48,874 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:48,875 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:48,875 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:29:58,891 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:29:58,891 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:29:58,914 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:29:58,917 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:29:58,917 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:08,930 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:30:08,931 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:30:08,947 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:08,950 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:08,950 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:18,963 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 15:30:18,963 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 15:30:18,986 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:18,989 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:18,989 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:29,003 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.85 % 2021-12-15 15:30:29,003 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 1.85 2021-12-15 15:30:29,038 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:29,040 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:29,040 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:39,052 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:30:39,052 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:30:39,072 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:39,075 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:39,075 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:49,086 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:30:49,088 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:30:49,102 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:49,105 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:49,105 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:30:59,117 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:30:59,117 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:30:59,137 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:30:59,140 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:30:59,140 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:09,151 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:09,151 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:09,171 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:09,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:09,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:19,188 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:19,188 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:19,213 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:19,216 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:19,216 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:29,229 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:29,229 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:29,246 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:29,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:29,250 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:39,263 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:39,263 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:39,286 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:39,289 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:39,290 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:49,305 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:49,305 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:49,322 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:49,325 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:49,325 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:31:59,348 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:31:59,349 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:31:59,367 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:31:59,369 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:31:59,369 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:09,381 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:09,381 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:09,398 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:09,401 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:09,403 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:19,418 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:19,418 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:19,436 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:19,440 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:19,440 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:29,450 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:29,450 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:29,467 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:29,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:29,470 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:39,480 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:39,482 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:39,499 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:39,502 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:39,502 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:49,513 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:49,513 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:49,529 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:49,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:49,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:32:59,542 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:32:59,542 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:32:59,558 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:32:59,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:32:59,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:09,571 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:33:09,571 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:33:09,588 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:09,589 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:09,591 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:19,602 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:33:19,602 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:33:19,618 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:19,619 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:19,619 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:29,631 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:33:29,631 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:33:29,647 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:29,648 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:29,648 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:39,660 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-15 15:33:39,661 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 7.41 2021-12-15 15:33:39,677 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:39,678 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:39,678 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:49,690 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.11 % 2021-12-15 15:33:49,690 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 11.11 2021-12-15 15:33:49,704 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:49,707 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:49,707 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:33:59,719 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:33:59,719 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:33:59,733 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:33:59,734 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:33:59,734 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:09,744 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:09,744 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:09,759 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:09,760 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:09,760 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:19,773 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:19,773 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:19,792 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:19,796 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:19,796 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:29,806 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:29,806 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:29,828 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:29,829 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:29,829 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:39,841 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:39,842 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:39,864 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:39,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:39,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:49,878 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:49,878 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:49,911 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:49,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:49,914 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:34:59,927 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:34:59,927 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:34:59,947 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:34:59,948 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:34:59,948 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:35:09,960 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:35:09,960 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:35:09,983 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:35:09,984 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:35:09,984 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:35:19,996 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:35:19,996 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:35:20,016 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:35:20,019 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:35:20,019 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:35:30,029 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:35:30,030 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:35:30,048 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:35:30,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:35:30,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:35:40,062 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:35:40,062 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:35:40,081 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:35:40,082 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:35:40,082 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:35:50,095 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:35:50,095 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:35:50,112 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:35:50,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:35:50,115 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:00,127 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:00,128 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:00,142 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:00,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:00,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:10,155 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:10,155 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:10,171 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:10,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:10,174 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:20,187 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:20,187 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:20,210 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:20,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:20,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:30,223 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:30,224 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:30,243 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:30,246 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:30,246 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:40,257 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:40,257 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:40,276 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:40,279 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:40,279 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:36:50,292 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:36:50,292 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:36:50,309 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:36:50,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:36:50,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:00,323 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.81 % 2021-12-15 15:37:00,325 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 14.81 2021-12-15 15:37:00,341 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:00,344 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:00,344 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:10,357 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.67 % 2021-12-15 15:37:10,357 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 16.67 2021-12-15 15:37:10,372 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:10,375 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:10,375 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:20,388 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:37:20,388 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:37:20,404 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:20,407 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:20,407 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:30,417 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:37:30,418 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:37:30,433 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:30,436 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:30,436 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:40,447 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:37:40,447 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:37:40,463 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:40,466 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:40,466 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:37:50,479 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:37:50,479 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:37:50,499 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:37:50,503 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:37:50,503 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:00,513 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:00,513 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:00,528 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:00,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:00,530 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:10,542 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:10,542 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:10,556 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:10,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:10,559 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:20,572 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:20,572 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:20,594 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:20,595 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:20,596 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:30,611 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:30,611 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:30,628 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:30,630 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:30,630 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:40,641 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:40,641 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:40,654 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:40,667 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:40,667 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:38:50,677 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:38:50,677 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:38:50,693 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:38:50,694 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:38:50,694 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:00,707 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:00,707 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:00,721 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:00,723 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:00,724 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:10,736 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:10,736 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:10,757 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:10,760 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:10,760 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:20,772 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:20,772 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:20,792 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:20,793 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:20,793 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:30,805 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:30,805 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:30,825 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:30,829 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:30,829 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:40,842 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:40,842 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:40,865 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:40,868 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:40,868 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:39:50,888 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:39:50,890 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:39:50,907 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:39:50,911 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:39:50,911 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:00,921 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:40:00,921 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:40:00,940 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:00,943 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:00,943 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:10,956 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:40:10,956 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:40:10,973 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:10,976 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:10,976 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:20,990 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:40:20,990 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:40:21,010 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:21,013 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:21,013 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:31,025 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-15 15:40:31,025 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-15 15:40:31,046 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:31,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:31,049 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:41,059 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.07 % 2021-12-15 15:40:41,061 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 24.07 2021-12-15 15:40:41,082 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:41,084 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:41,084 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:40:51,095 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:40:51,095 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:40:51,111 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:40:51,114 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:40:51,114 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:01,125 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:01,125 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:01,144 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:01,147 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:01,147 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:11,157 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:11,157 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:11,178 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:11,180 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:11,180 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:21,191 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:21,193 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:21,210 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:21,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:21,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:31,223 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:31,223 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:31,240 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:31,243 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:31,243 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:41,255 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:41,255 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:41,273 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:41,276 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:41,276 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:41:51,288 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:41:51,289 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:41:51,309 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:41:51,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:41:51,312 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:01,323 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:01,323 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:01,344 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:01,346 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:01,346 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:11,358 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:11,358 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:11,375 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:11,378 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:11,378 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:21,391 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:21,391 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:21,407 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:21,410 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:21,410 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:31,421 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:31,423 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:31,440 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:31,444 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:31,444 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:41,456 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:41,456 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:41,476 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:41,480 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:41,480 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:42:51,490 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:42:51,490 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:42:51,506 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:42:51,509 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:42:51,509 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:01,519 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:43:01,519 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:43:01,535 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:01,538 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:01,538 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:11,549 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:43:11,551 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:43:11,568 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:11,572 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:11,572 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:21,584 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:43:21,584 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:43:21,598 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:21,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:21,601 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:31,618 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:43:31,619 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:43:31,635 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:31,638 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:31,638 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:41,654 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.63 % 2021-12-15 15:43:41,654 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 29.63 2021-12-15 15:43:41,674 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:41,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:41,677 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:43:51,690 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.48 % 2021-12-15 15:43:51,690 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 31.48 2021-12-15 15:43:51,707 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:43:51,710 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:43:51,710 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:01,723 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.19 % 2021-12-15 15:44:01,723 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 35.19 2021-12-15 15:44:01,739 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:01,742 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:01,742 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:11,753 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:44:11,753 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:44:11,770 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:11,773 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:11,773 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:21,788 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:44:21,789 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:44:21,802 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:21,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:21,805 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:31,816 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:44:31,816 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:44:31,832 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:31,835 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:31,835 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:41,848 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:44:41,848 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:44:41,864 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:41,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:41,865 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:44:51,878 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:44:51,878 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:44:51,894 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:44:51,897 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:44:51,897 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:01,907 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:01,907 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:01,937 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:01,938 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:01,940 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:11,950 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:11,950 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:11,964 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:11,967 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:11,967 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:21,979 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:21,979 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:22,000 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:22,002 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:22,002 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:32,013 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:32,013 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:32,036 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:32,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:32,039 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:42,052 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:42,052 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:42,069 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:42,072 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:42,072 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:45:52,084 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:45:52,084 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:45:52,105 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:45:52,107 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:45:52,107 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:02,121 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:02,122 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:02,141 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:02,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:02,144 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:12,154 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:12,154 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:12,176 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:12,177 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:12,177 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:22,190 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:22,190 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:22,210 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:22,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:22,213 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:32,226 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:32,226 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:32,243 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:32,244 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:32,244 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:42,257 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:42,259 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:42,275 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:42,276 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:42,276 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:46:52,292 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.04 % 2021-12-15 15:46:52,292 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 37.04 2021-12-15 15:46:52,315 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:46:52,318 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:46:52,318 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:02,328 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.89 % 2021-12-15 15:47:02,329 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 38.89 2021-12-15 15:47:02,342 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:02,345 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:02,345 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:12,359 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-15 15:47:12,359 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-15 15:47:12,380 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:12,382 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:12,382 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:22,397 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:47:22,397 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:47:22,415 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:22,417 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:22,418 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:32,431 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:47:32,433 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:47:32,450 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:32,451 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:32,453 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:42,463 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:47:42,463 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:47:42,483 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:42,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:42,486 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:47:52,497 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:47:52,497 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:47:52,515 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:47:52,517 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:47:52,517 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:02,530 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:02,530 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:02,562 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:02,566 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:02,566 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:12,578 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:12,578 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:12,594 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:12,598 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:12,598 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:22,608 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:22,609 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:22,628 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:22,631 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:22,631 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:32,644 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:32,644 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:32,661 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:32,664 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:32,664 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:42,676 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:42,676 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:42,693 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:42,696 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:42,696 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:48:52,707 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:48:52,707 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:48:52,721 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:48:52,724 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:48:52,724 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:49:02,734 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:49:02,736 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:49:02,750 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:49:02,753 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:49:02,753 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:49:12,765 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:49:12,766 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:49:12,785 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:49:12,788 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:49:12,789 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:49:22,801 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.44 % 2021-12-15 15:49:22,802 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 44.44 2021-12-15 15:49:22,819 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:49:22,822 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:49:22,822 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:49:32,835 | test | DEBUG | pool-3-thread-24 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.30 % 2021-12-15 15:49:32,835 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: running, progress: 46.3 2021-12-15 15:49:32,851 | infra | DEBUG | pool-3-thread-24 | [task:check:412] Wait before next rebalance progress check 2021-12-15 15:49:32,855 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 15:49:32,855 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 15:49:42,874 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-15 15:49:42,874 | infra | DEBUG | pool-3-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-15 15:49:47,888 | test | INFO | pool-3-thread-24 | [task:check:385] Rebalance - status: none, progress: 100 2021-12-15 15:49:47,907 | test | INFO | pool-3-thread-24 | [task:check:444] Rebalance completed with progress: 100% in 1359.62700009 sec 2021-12-15 15:49:47,907 | infra | INFO | pool-3-thread-24 | [task:complete_task:105] Thread 'Rebalance_task_IN=[]_OUT=[]_1639610828.15' completed 2021-12-15 15:49:47,946 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 49774.90000000001, u'diskFetches': 239165004, u'dataUsed': 7973769640191L, u'itemCount': 3999987961L, u'quotaPercentUsed': 79.5905285384506, u'storageTotals': {u'hdd': {u'usedByData': 12359138755562L, u'used': 12824667509222L, u'quotaTotal': 47967746654208L, u'total': 47967746654208L, u'free': 34106304259692L}, u'ram': {u'usedByData': 341838717144L, u'used': 939584630784L, u'quotaTotal': 776848015360L, u'total': 1079482892288L, u'quotaUsed': 429496729600L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 12359138755562L, u'vbActiveNumNonResident': 3820333776L, u'memUsed': 341838717144L} 2021-12-15 15:49:48,173 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+--------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+--------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999987961 | 400.00 GiB | 318.36 GiB | 11.24 TiB | 4.4913681429 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+--------------+ 2021-12-15 15:49:48,174 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-15 15:49:48,516 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 4.49290677712 2021-12-15 15:49:48,517 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:49:48,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:49:48,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:49:48,842 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:49:48,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:49:48,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:49:48,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:49:48,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:49:48,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:49:48,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:49:48,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:49:48,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 15:49:49,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 15:49:49,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 15:49:49,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 15:49:49,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 419 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 402 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-504 is 417 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 336 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-360 is 359 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 389 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 410 2021-12-15 15:49:53,019 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 396 2021-12-15 15:49:53,020 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-793 is 374 2021-12-15 15:49:53,020 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 379 2021-12-15 15:49:53,020 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 374 2021-12-15 15:49:53,020 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-433 is 379 2021-12-15 15:49:53,020 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-361 is 417 2021-12-15 15:49:53,022 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 353 2021-12-15 15:49:53,022 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 422 2021-12-15 15:49:53,022 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 389 2021-12-15 15:49:53,022 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-730 is 337 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 304 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-370 is 367 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-226 is 401 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 359 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 400 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 340 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-731 is 359 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 341 2021-12-15 15:49:53,023 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 397 2021-12-15 15:49:53,025 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 352 2021-12-15 15:49:53,025 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 359 2021-12-15 15:49:53,025 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 390 2021-12-15 15:49:53,025 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-732 is 317 2021-12-15 15:49:53,025 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-444 is 412 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-372 is 384 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 337 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 374 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 309 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 335 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-733 is 300 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 324 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-373 is 355 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 365 2021-12-15 15:49:53,026 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 303 2021-12-15 15:49:53,028 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 336 2021-12-15 15:49:53,028 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-806 is 378 2021-12-15 15:49:53,028 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 388 2021-12-15 15:49:53,028 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 357 2021-12-15 15:49:53,028 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-374 is 429 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 344 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 347 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 368 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-735 is 330 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-447 is 315 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-375 is 342 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-231 is 318 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 311 2021-12-15 15:49:53,029 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 314 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-736 is 318 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 340 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-376 is 350 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 355 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 329 2021-12-15 15:49:53,030 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-809 is 317 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-737 is 363 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 303 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-377 is 361 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 306 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 357 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-810 is 354 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-738 is 366 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 339 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-378 is 327 2021-12-15 15:49:53,032 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 317 2021-12-15 15:49:53,033 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 373 2021-12-15 15:49:53,033 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 363 2021-12-15 15:49:53,033 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 357 2021-12-15 15:49:53,033 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 395 2021-12-15 15:49:53,033 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 335 2021-12-15 15:49:53,035 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 367 2021-12-15 15:49:53,035 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-794 is 367 2021-12-15 15:49:53,035 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 389 2021-12-15 15:49:53,035 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 367 2021-12-15 15:49:53,035 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 379 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-218 is 426 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-74 is 311 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 346 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-812 is 372 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-740 is 367 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 307 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 379 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 340 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 341 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 301 2021-12-15 15:49:53,036 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-741 is 352 2021-12-15 15:49:53,038 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 354 2021-12-15 15:49:53,038 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-381 is 358 2021-12-15 15:49:53,038 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 354 2021-12-15 15:49:53,038 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 336 2021-12-15 15:49:53,038 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 323 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-742 is 359 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-454 is 338 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 375 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 343 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 321 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-743 is 340 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-455 is 346 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 340 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 368 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-816 is 381 2021-12-15 15:49:53,039 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-744 is 356 2021-12-15 15:49:53,040 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 402 2021-12-15 15:49:53,040 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-384 is 320 2021-12-15 15:49:53,040 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 310 2021-12-15 15:49:53,040 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 375 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-817 is 332 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-745 is 351 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-457 is 340 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-385 is 335 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-241 is 340 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 301 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 353 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-746 is 331 2021-12-15 15:49:53,042 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 315 2021-12-15 15:49:53,043 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-386 is 333 2021-12-15 15:49:53,043 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 391 2021-12-15 15:49:53,043 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-26 is 365 2021-12-15 15:49:53,043 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-819 is 317 2021-12-15 15:49:53,043 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-747 is 379 2021-12-15 15:49:53,045 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-459 is 383 2021-12-15 15:49:53,045 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-387 is 358 2021-12-15 15:49:53,045 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 370 2021-12-15 15:49:53,045 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 390 2021-12-15 15:49:53,045 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 373 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-820 is 386 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-748 is 345 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-388 is 392 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 345 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-28 is 306 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 312 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 324 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 338 2021-12-15 15:49:53,046 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 359 2021-12-15 15:49:53,048 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 373 2021-12-15 15:49:53,048 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 349 2021-12-15 15:49:53,048 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-723 is 391 2021-12-15 15:49:53,048 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 375 2021-12-15 15:49:53,048 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 351 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-363 is 415 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 360 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-75 is 327 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 395 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 409 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 354 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-390 is 363 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-246 is 381 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-174 is 387 2021-12-15 15:49:53,049 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 379 2021-12-15 15:49:53,051 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-823 is 379 2021-12-15 15:49:53,051 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-751 is 310 2021-12-15 15:49:53,051 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 367 2021-12-15 15:49:53,051 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 355 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-247 is 388 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 348 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 306 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 351 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-752 is 335 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 374 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 366 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 338 2021-12-15 15:49:53,052 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 314 2021-12-15 15:49:53,053 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 369 2021-12-15 15:49:53,053 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 355 2021-12-15 15:49:53,053 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-753 is 364 2021-12-15 15:49:53,053 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-465 is 329 2021-12-15 15:49:53,053 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-393 is 388 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 406 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-177 is 407 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 373 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 348 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-754 is 379 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 374 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-394 is 383 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 398 2021-12-15 15:49:53,055 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 371 2021-12-15 15:49:53,056 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 366 2021-12-15 15:49:53,056 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-755 is 348 2021-12-15 15:49:53,056 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-683 is 333 2021-12-15 15:49:53,056 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 329 2021-12-15 15:49:53,058 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-395 is 367 2021-12-15 15:49:53,058 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 358 2021-12-15 15:49:53,058 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 319 2021-12-15 15:49:53,058 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 317 2021-12-15 15:49:53,058 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 388 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 328 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-684 is 370 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 338 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-396 is 373 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-252 is 398 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 348 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 428 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-757 is 370 2021-12-15 15:49:53,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-397 is 363 2021-12-15 15:49:53,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 360 2021-12-15 15:49:53,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-181 is 324 2021-12-15 15:49:53,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 409 2021-12-15 15:49:53,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-830 is 318 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 372 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 354 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 399 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-398 is 423 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 354 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 403 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 373 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 391 2021-12-15 15:49:53,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-687 is 328 2021-12-15 15:49:53,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-471 is 405 2021-12-15 15:49:53,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-399 is 416 2021-12-15 15:49:53,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 315 2021-12-15 15:49:53,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 374 2021-12-15 15:49:53,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-796 is 417 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-724 is 397 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-508 is 386 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 398 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-364 is 353 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 392 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 411 2021-12-15 15:49:53,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 364 2021-12-15 15:49:53,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 329 2021-12-15 15:49:53,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 332 2021-12-15 15:49:53,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 334 2021-12-15 15:49:53,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 320 2021-12-15 15:49:53,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-40 is 344 2021-12-15 15:49:53,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-833 is 363 2021-12-15 15:49:53,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-761 is 391 2021-12-15 15:49:53,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-689 is 343 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-473 is 395 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-401 is 383 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 323 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 368 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 351 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-762 is 406 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-690 is 409 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 382 2021-12-15 15:49:53,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 345 2021-12-15 15:49:53,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 395 2021-12-15 15:49:53,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 383 2021-12-15 15:49:53,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-835 is 369 2021-12-15 15:49:53,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-763 is 313 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-691 is 414 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 374 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-403 is 336 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-187 is 353 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 362 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 362 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-764 is 391 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-692 is 332 2021-12-15 15:49:53,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 337 2021-12-15 15:49:53,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 327 2021-12-15 15:49:53,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 350 2021-12-15 15:49:53,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 308 2021-12-15 15:49:53,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-837 is 366 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-765 is 347 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-693 is 401 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 350 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-405 is 300 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 339 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 332 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-838 is 317 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-766 is 330 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-694 is 350 2021-12-15 15:49:53,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 355 2021-12-15 15:49:53,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 344 2021-12-15 15:49:53,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 342 2021-12-15 15:49:53,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 373 2021-12-15 15:49:53,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 396 2021-12-15 15:49:53,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 370 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-695 is 384 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-479 is 389 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 311 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-191 is 386 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-47 is 325 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 334 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-696 is 403 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 364 2021-12-15 15:49:53,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-408 is 393 2021-12-15 15:49:53,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-192 is 383 2021-12-15 15:49:53,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-48 is 382 2021-12-15 15:49:53,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 301 2021-12-15 15:49:53,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-769 is 392 2021-12-15 15:49:53,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-697 is 379 2021-12-15 15:49:53,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-481 is 348 2021-12-15 15:49:53,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 355 2021-12-15 15:49:53,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 345 2021-12-15 15:49:53,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 316 2021-12-15 15:49:53,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 358 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-725 is 344 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 344 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 384 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-365 is 360 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 363 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 355 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 365 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 400 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 377 2021-12-15 15:49:53,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 349 2021-12-15 15:49:53,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 369 2021-12-15 15:49:53,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-410 is 319 2021-12-15 15:49:53,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 382 2021-12-15 15:49:53,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 339 2021-12-15 15:49:53,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-843 is 378 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-771 is 371 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-699 is 369 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 326 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-411 is 414 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 335 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 353 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-772 is 308 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-700 is 329 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-484 is 316 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-412 is 380 2021-12-15 15:49:53,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-196 is 363 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-52 is 364 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 304 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-773 is 363 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 352 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-485 is 368 2021-12-15 15:49:53,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 356 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 391 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 320 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 344 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-774 is 392 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-702 is 375 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 338 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 345 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 348 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 366 2021-12-15 15:49:53,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-847 is 368 2021-12-15 15:49:53,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-775 is 376 2021-12-15 15:49:53,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-703 is 330 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 303 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-415 is 397 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 382 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 331 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 318 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 372 2021-12-15 15:49:53,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 316 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 401 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-344 is 333 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 351 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 420 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 344 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-777 is 346 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-705 is 326 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 414 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-417 is 386 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 342 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 378 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 375 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 371 2021-12-15 15:49:53,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-778 is 365 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-706 is 353 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 370 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 367 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 379 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 371 2021-12-15 15:49:53,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 415 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 432 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 396 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 357 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-419 is 439 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 347 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 355 2021-12-15 15:49:53,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 381 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 376 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-726 is 359 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 398 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 331 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-366 is 362 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-222 is 360 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 347 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-6 is 324 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-852 is 356 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 340 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-708 is 358 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 366 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-420 is 357 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-348 is 356 2021-12-15 15:49:53,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 341 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 348 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 352 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-709 is 342 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 339 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 329 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 368 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-205 is 347 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-782 is 379 2021-12-15 15:49:53,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 343 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 424 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-422 is 372 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 335 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 372 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 379 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 351 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 390 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-423 is 347 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 349 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-207 is 349 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-63 is 360 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-784 is 372 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-712 is 348 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-496 is 394 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-424 is 401 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-352 is 355 2021-12-15 15:49:53,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 344 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-64 is 368 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-785 is 395 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-713 is 332 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 384 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-425 is 392 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 313 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-65 is 385 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 347 2021-12-15 15:49:53,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-714 is 363 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-498 is 385 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-426 is 319 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-354 is 321 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 344 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 327 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 404 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-715 is 391 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 331 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-427 is 333 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-355 is 368 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 343 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 367 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 347 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-716 is 340 2021-12-15 15:49:53,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 379 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-428 is 359 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 308 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 348 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 387 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-789 is 319 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-717 is 392 2021-12-15 15:49:53,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 319 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 348 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-213 is 351 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 308 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 400 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-727 is 415 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 414 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 365 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-367 is 346 2021-12-15 15:49:53,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 348 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-79 is 435 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 405 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-790 is 354 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-718 is 383 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 380 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 335 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-358 is 355 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 398 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 382 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 353 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-719 is 367 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 369 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 335 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-359 is 387 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 386 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 328 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 383 2021-12-15 15:49:53,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-728 is 393 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 447 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 344 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-368 is 445 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 415 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 447 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 410 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-801 is 364 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-729 is 389 2021-12-15 15:49:53,117 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 322 2021-12-15 15:49:53,118 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-369 is 353 2021-12-15 15:49:53,118 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 387 2021-12-15 15:49:53,118 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 342 2021-12-15 15:49:53,118 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 366 2021-12-15 15:49:53,118 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:49:53,121 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:49:53,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:49:53,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:49:53,559 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:49:53,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:49:53,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:49:53,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:49:53,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:49:53,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:49:53,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:49:53,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:49:53,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 15:49:54,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 15:49:54,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 15:49:54,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 15:49:54,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-864 is 317 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-576 is 385 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-1009 is 339 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-937 is 389 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 417 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 313 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-82 is 310 2021-12-15 15:49:57,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-875 is 303 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 335 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 303 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-371 is 326 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-83 is 327 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-11 is 316 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-876 is 322 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 312 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-372 is 319 2021-12-15 15:49:57,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-84 is 424 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-12 is 332 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-877 is 300 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-589 is 309 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-517 is 301 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 311 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-878 is 309 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-590 is 313 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-518 is 410 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 307 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-879 is 399 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-591 is 325 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-15 is 342 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-880 is 312 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-520 is 318 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-160 is 325 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-16 is 308 2021-12-15 15:49:57,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-593 is 307 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-377 is 318 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-954 is 303 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-882 is 312 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-594 is 321 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-522 is 362 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-162 is 320 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-883 is 315 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-379 is 304 2021-12-15 15:49:57,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 300 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-19 is 303 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-938 is 355 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-866 is 322 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-578 is 303 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-362 is 379 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-74 is 419 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-956 is 300 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-20 is 349 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-957 is 328 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-597 is 379 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-165 is 317 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-93 is 301 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-21 is 306 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-958 is 328 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-382 is 338 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-166 is 328 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-94 is 322 2021-12-15 15:49:57,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-887 is 335 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-167 is 322 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-960 is 321 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-888 is 301 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-384 is 378 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-96 is 302 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-169 is 308 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-97 is 317 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-25 is 317 2021-12-15 15:49:57,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-962 is 301 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-170 is 313 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-26 is 314 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-531 is 310 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-387 is 383 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-99 is 323 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-388 is 311 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-100 is 303 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-28 is 323 2021-12-15 15:49:57,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-965 is 402 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-893 is 325 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-533 is 321 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-389 is 332 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-101 is 313 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-363 is 328 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-75 is 312 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-966 is 308 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-390 is 300 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-30 is 365 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-895 is 320 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-391 is 358 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-968 is 396 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-896 is 300 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-536 is 309 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-392 is 307 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-32 is 317 2021-12-15 15:49:57,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-537 is 346 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 306 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-394 is 305 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-539 is 313 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-107 is 312 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-972 is 396 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-900 is 303 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-396 is 314 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-973 is 304 2021-12-15 15:49:57,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-109 is 332 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-974 is 324 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-398 is 309 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-111 is 336 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-39 is 326 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-148 is 305 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-976 is 307 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-544 is 365 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-977 is 344 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-905 is 356 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-545 is 336 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-906 is 300 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-114 is 327 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-42 is 309 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-979 is 321 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-403 is 325 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-115 is 331 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-980 is 301 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 321 2021-12-15 15:49:57,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-116 is 356 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-44 is 303 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-910 is 414 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-46 is 322 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-407 is 337 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 326 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-984 is 327 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 374 2021-12-15 15:49:57,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-48 is 317 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-985 is 300 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-553 is 321 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-941 is 320 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-581 is 319 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-5 is 315 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-122 is 315 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-915 is 317 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-555 is 357 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 420 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-52 is 314 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-557 is 318 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-125 is 300 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-414 is 336 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-415 is 311 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-343 is 303 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-127 is 313 2021-12-15 15:49:57,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-992 is 315 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-128 is 308 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-56 is 306 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-993 is 312 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 310 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-417 is 305 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-345 is 378 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-418 is 318 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-58 is 326 2021-12-15 15:49:57,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-563 is 305 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 303 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-347 is 380 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-870 is 306 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-366 is 314 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-6 is 303 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-996 is 324 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-924 is 303 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-564 is 312 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-348 is 301 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-60 is 324 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-61 is 326 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-422 is 326 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-350 is 313 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-62 is 306 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-999 is 304 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-135 is 315 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-1000 is 321 2021-12-15 15:49:57,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-136 is 302 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-64 is 305 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-1001 is 300 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-929 is 324 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-569 is 320 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-353 is 318 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-137 is 305 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-65 is 313 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-354 is 341 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-138 is 312 2021-12-15 15:49:57,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-66 is 327 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-859 is 318 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-427 is 309 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-355 is 409 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-1004 is 393 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-572 is 301 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-356 is 312 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-140 is 331 2021-12-15 15:49:57,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-68 is 325 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-933 is 329 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-861 is 316 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-573 is 319 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-69 is 326 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-583 is 337 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-151 is 320 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-79 is 313 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-934 is 331 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-574 is 305 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-142 is 390 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-70 is 317 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-1007 is 300 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-359 is 324 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-143 is 302 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-944 is 409 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-872 is 373 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-368 is 312 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 360 2021-12-15 15:49:57,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-1017 is 342 2021-12-15 15:49:57,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-945 is 316 2021-12-15 15:49:57,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-585 is 331 2021-12-15 15:49:57,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-153 is 326 2021-12-15 15:49:57,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-9 is 305 2021-12-15 15:49:57,107 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:49:57,109 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:49:57,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:49:57,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:49:57,533 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:49:57,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:49:57,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:49:57,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:49:57,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:49:57,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:49:57,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:49:57,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:49:57,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 15:49:57,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 15:49:57,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 15:49:58,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 15:49:58,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 15:50:00,141 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.17/kvstore-161 is 317 2021-12-15 15:50:00,141 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.17/kvstore-305 is 346 2021-12-15 15:50:00,141 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.29/kvstore-677 is 311 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.46/kvstore-622 is 306 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.57/kvstore-489 is 356 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.6/kvstore-1014 is 310 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.61/kvstore-133 is 323 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.65/kvstore-497 is 301 2021-12-15 15:50:00,142 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.68--/data/GleamBookUsers0/magma.68/kvstore-644 is 303 2021-12-15 15:50:00,144 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:50:00,145 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:50:00,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:50:00,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:50:00,631 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:50:00,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:50:00,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:50:00,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:50:00,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:50:00,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:50:00,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:50:00,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:50:00,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 15:50:01,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 15:50:01,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 15:50:01,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 15:50:01,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 15:50:04,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-936 is 321 2021-12-15 15:50:04,351 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-648 is 327 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-576 is 308 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-937 is 319 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-865 is 318 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-649 is 333 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-289 is 306 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-217 is 308 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-730 is 324 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-658 is 314 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-586 is 316 2021-12-15 15:50:04,352 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-298 is 309 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-226 is 307 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.11/kvstore-875 is 308 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.11/kvstore-587 is 303 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.11/kvstore-227 is 304 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-876 is 324 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-660 is 376 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-516 is 329 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-300 is 317 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-733 is 317 2021-12-15 15:50:04,354 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-517 is 309 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-229 is 303 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-734 is 421 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-662 is 325 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-230 is 302 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-1023 is 318 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-879 is 319 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-735 is 355 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-303 is 326 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-880 is 309 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-736 is 302 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-664 is 328 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-592 is 316 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-881 is 301 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-737 is 307 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-665 is 345 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-521 is 301 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-233 is 326 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-882 is 345 2021-12-15 15:50:04,355 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-306 is 318 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-883 is 348 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-739 is 311 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-667 is 340 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-595 is 302 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-884 is 314 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-740 is 348 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-668 is 314 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-885 is 314 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-597 is 311 2021-12-15 15:50:04,357 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-309 is 307 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-237 is 325 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-742 is 315 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-887 is 301 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-311 is 300 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-888 is 301 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-672 is 368 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-600 is 319 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-528 is 300 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-312 is 332 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-240 is 307 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-673 is 301 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-313 is 319 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-241 is 326 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-890 is 319 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-746 is 435 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-314 is 305 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-242 is 304 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-891 is 382 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-747 is 302 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-243 is 308 2021-12-15 15:50:04,358 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-532 is 316 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-244 is 312 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-172 is 301 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-893 is 317 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-677 is 322 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-605 is 300 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-533 is 392 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-317 is 318 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-867 is 323 2021-12-15 15:50:04,359 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-723 is 300 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-651 is 318 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-579 is 307 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-750 is 334 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-678 is 304 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-606 is 415 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-534 is 315 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-318 is 303 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-174 is 301 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-895 is 321 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-535 is 305 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-247 is 341 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-752 is 324 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-680 is 305 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-536 is 324 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-753 is 303 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-681 is 324 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-898 is 309 2021-12-15 15:50:04,361 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-682 is 328 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-610 is 448 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-250 is 316 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-178 is 342 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-899 is 331 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-251 is 303 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-900 is 310 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-684 is 373 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-612 is 312 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-324 is 302 2021-12-15 15:50:04,362 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-180 is 312 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-901 is 408 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-613 is 305 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-325 is 309 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-181 is 302 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-326 is 315 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-182 is 331 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-903 is 328 2021-12-15 15:50:04,364 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-687 is 309 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-615 is 308 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-183 is 319 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-868 is 311 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-220 is 302 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-760 is 311 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-688 is 315 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-616 is 311 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-256 is 304 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-905 is 300 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-617 is 328 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-545 is 317 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-257 is 333 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-185 is 318 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.42/kvstore-762 is 302 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-691 is 309 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-619 is 334 2021-12-15 15:50:04,365 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-547 is 325 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-331 is 307 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-187 is 321 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.44/kvstore-620 is 323 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-765 is 321 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-693 is 390 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-621 is 312 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-549 is 328 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-910 is 301 2021-12-15 15:50:04,367 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-622 is 321 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-334 is 310 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-262 is 324 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-767 is 310 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-551 is 324 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-335 is 366 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-912 is 470 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-696 is 305 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-624 is 323 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-552 is 305 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-192 is 300 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-913 is 313 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-625 is 411 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-553 is 433 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-265 is 313 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-581 is 332 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-914 is 313 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-698 is 317 2021-12-15 15:50:04,368 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-626 is 321 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-338 is 319 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-266 is 365 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-194 is 301 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-339 is 319 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-195 is 301 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-916 is 313 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-700 is 331 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-340 is 300 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-268 is 318 2021-12-15 15:50:04,369 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-196 is 327 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-701 is 429 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-269 is 309 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-197 is 317 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-918 is 313 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-558 is 321 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-270 is 310 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-198 is 315 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-919 is 349 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-703 is 336 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-559 is 313 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-199 is 307 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-200 is 330 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-705 is 332 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-633 is 302 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-561 is 324 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-273 is 305 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.58/kvstore-562 is 307 2021-12-15 15:50:04,371 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.58/kvstore-274 is 310 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.58/kvstore-202 is 302 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.59/kvstore-707 is 307 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.59/kvstore-635 is 304 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.59/kvstore-275 is 308 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-870 is 301 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-654 is 330 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-924 is 313 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-708 is 317 2021-12-15 15:50:04,372 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-564 is 342 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-276 is 305 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-204 is 326 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-709 is 400 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-565 is 427 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-277 is 305 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-854 is 362 2021-12-15 15:50:04,374 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-710 is 326 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-638 is 303 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-566 is 300 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-206 is 312 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-711 is 323 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-279 is 312 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-568 is 339 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-208 is 302 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-929 is 310 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-713 is 352 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-569 is 373 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-281 is 314 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-209 is 305 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-570 is 313 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-210 is 310 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-931 is 327 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-859 is 342 2021-12-15 15:50:04,375 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-715 is 332 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-643 is 332 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-571 is 315 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-644 is 343 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-572 is 314 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-284 is 315 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-933 is 318 2021-12-15 15:50:04,377 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-717 is 326 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-645 is 322 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-573 is 334 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-285 is 309 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-213 is 304 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-727 is 317 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-583 is 321 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-295 is 320 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-223 is 315 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-935 is 326 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-863 is 301 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-728 is 316 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-584 is 325 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-224 is 396 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-729 is 306 2021-12-15 15:50:04,378 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-585 is 325 2021-12-15 15:50:04,378 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:50:04,378 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 11827937MB 2021-12-15 15:50:04,380 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 10397MB 2021-12-15 15:50:04,380 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 384576MB 2021-12-15 15:50:04,380 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 11442910MB 2021-12-15 15:50:04,382 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:50:04,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:50:04,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:50:04,775 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:50:04,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:50:04,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:50:04,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:50:04,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:50:04,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:50:04,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:50:04,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:50:04,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 15:50:04,913 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-15 15:50:04,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 15:50:05,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 15:50:05,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 15:50:06,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 15:50:06,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 15:50:06,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 15:50:07,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 15:50:07,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 15:50:07,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 15:50:08,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 15:50:08,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 15:50:09,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 15:50:09,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 15:50:09,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 15:50:10,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 15:50:10,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 15:50:10,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 15:50:11,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 15:50:11,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 15:50:11,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 15:50:12,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 15:50:12,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 15:50:12,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 15:50:13,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 15:50:13,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 15:50:14,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 15:50:14,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 15:50:14,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 15:50:15,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 15:50:15,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 15:50:16,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 15:50:16,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 15:50:16,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 15:50:17,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 15:50:17,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 15:50:17,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 15:50:18,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 15:50:18,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 15:50:19,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 15:50:19,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 15:50:19,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 15:50:20,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 15:50:20,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 15:50:20,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 15:50:21,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 15:50:21,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 15:50:21,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 15:50:22,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 15:50:22,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 15:50:22,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 15:50:23,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 15:50:23,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 15:50:24,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 15:50:24,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 15:50:24,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 15:50:25,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 15:50:25,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 15:50:25,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 15:50:26,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 15:50:26,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 15:50:26,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 15:50:27,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 15:50:27,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 15:50:28,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 15:50:28,292 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:50:28,296 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:50:28,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:50:28,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:50:28,729 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:50:28,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:50:28,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:50:28,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:50:28,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:50:28,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:50:28,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:50:28,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:50:28,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 15:50:28,868 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-15 15:50:28,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 15:50:29,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 15:50:29,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 15:50:29,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 15:50:30,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 15:50:30,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 15:50:31,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 15:50:31,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 15:50:31,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 15:50:32,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 15:50:32,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 15:50:32,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 15:50:33,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 15:50:33,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 15:50:33,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 15:50:34,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 15:50:34,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 15:50:35,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 15:50:35,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 15:50:35,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 15:50:36,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 15:50:36,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 15:50:37,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 15:50:37,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 15:50:37,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 15:50:38,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 15:50:38,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 15:50:38,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 15:50:39,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 15:50:39,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 15:50:39,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 15:50:40,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 15:50:40,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 15:50:40,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 15:50:41,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 15:50:41,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 15:50:42,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 15:50:42,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 15:50:42,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 15:50:43,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 15:50:43,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 15:50:43,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 15:50:44,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 15:50:44,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 15:50:44,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 15:50:45,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 15:50:45,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 15:50:46,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 15:50:46,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 15:50:46,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 15:50:47,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 15:50:47,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 15:50:47,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 15:50:48,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 15:50:48,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 15:50:49,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 15:50:49,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 15:50:49,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 15:50:50,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 15:50:50,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 15:50:51,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 15:50:51,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 15:50:51,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 15:50:52,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 15:50:52,684 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:50:52,688 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:50:52,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:50:53,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:50:53,056 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:50:53,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:50:53,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:50:53,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:50:53,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:50:53,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:50:53,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:50:53,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:50:53,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 15:50:53,194 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.68 - core(s): 72 2021-12-15 15:50:53,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 15:50:53,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 15:50:54,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 15:50:54,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 15:50:54,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 15:50:55,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 15:50:55,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 15:50:55,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 15:50:56,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 15:50:56,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 15:50:56,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 15:50:57,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 15:50:57,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 15:50:57,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 15:50:58,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 15:50:58,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 15:50:59,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 15:50:59,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 15:50:59,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 15:51:00,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 15:51:00,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 15:51:00,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 15:51:01,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 15:51:01,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 15:51:01,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 15:51:02,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 15:51:02,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 15:51:03,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 15:51:03,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 15:51:03,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 15:51:04,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 15:51:04,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 15:51:04,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 15:51:05,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 15:51:05,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 15:51:05,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 15:51:06,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 15:51:06,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 15:51:07,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 15:51:07,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 15:51:07,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 15:51:08,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 15:51:08,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 15:51:08,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 15:51:09,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 15:51:09,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 15:51:09,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 15:51:10,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 15:51:10,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 15:51:11,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 15:51:11,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 15:51:11,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 15:51:12,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 15:51:12,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 15:51:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 15:51:13,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 15:51:13,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 15:51:13,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 15:51:14,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 15:51:14,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 15:51:14,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 15:51:15,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 15:51:15,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 15:51:16,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 15:51:16,369 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:51:16,372 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:51:16,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:51:16,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:51:16,736 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:51:16,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:51:16,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:51:16,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:51:16,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:51:16,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:51:16,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:51:16,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:51:16,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 15:51:16,891 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-15 15:51:16,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 15:51:17,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 15:51:17,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 15:51:18,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 15:51:18,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 15:51:18,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 15:51:18,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 15:51:19,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 15:51:19,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 15:51:20,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 15:51:20,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 15:51:20,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 15:51:21,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 15:51:21,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 15:51:21,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 15:51:22,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 15:51:22,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 15:51:22,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 15:51:23,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 15:51:23,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 15:51:23,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 15:51:24,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 15:51:24,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 15:51:25,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 15:51:25,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 15:51:25,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 15:51:26,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 15:51:26,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 15:51:26,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 15:51:27,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 15:51:27,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 15:51:27,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 15:51:28,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 15:51:28,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 15:51:29,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 15:51:29,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 15:51:29,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 15:51:30,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 15:51:30,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 15:51:30,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 15:51:31,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 15:51:31,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 15:51:32,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 15:51:32,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 15:51:32,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 15:51:32,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 15:51:33,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 15:51:33,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 15:51:34,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 15:51:34,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 15:51:34,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 15:51:35,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 15:51:35,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 15:51:35,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 15:51:36,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 15:51:36,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 15:51:37,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 15:51:37,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 15:51:37,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 15:51:38,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 15:51:38,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 15:51:38,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 15:51:39,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 15:51:39,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 15:51:39,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:51:39,953 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 600042.6875, u'ReplayWALOffset': u'10118:13189120', u'NTablesCreated': 606126, u'RecentReadIOAmpSet': 0.1985311508178711, u'ReadIOAmp': 2619.496826171875, u'BlockCacheMisses': 60658368, u'NFlushes': 144465, u'RecentWriteAmp': 2.452514171600342, u'NumSyncFlushes': 2206, u'TotalMemUsed': 138816246, u'NBloomFilterHits': 674815509, u'Fragmentation': 0.41140368580818176, u'NumNormalFlushes': 78305, u'RecentBlockCacheHitRatio': 86.57295989990234, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 324931, u'ReadIOAmp': 1.0122625292924856, u'MemtablesMemUsed': 8165952, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 360391029, u'CheckpointOverhead': 6742016, u'NItemsGet': 874371, u'NWriteIOs': 491141, u'NReadBytes': 32373335238L, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 17052218, u'SizeRatio': 0.508195698261261, u'NWriteBytes': 945041408, u'BloomFilterMemUsed': 0, u'NumTables': 72, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 815104}, {u'TargetSize': 33554432, u'LogicalSize': 25688835, u'SizeRatio': 0.7655869424343109, u'NWriteBytes': 784470016, u'BloomFilterMemUsed': 0, u'NumTables': 15, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 823296}, {u'TargetSize': 485151710, u'LogicalSize': 393011268, u'SizeRatio': 0.8100791152524228, u'NWriteBytes': 2863710208L, u'BloomFilterMemUsed': 0, u'NumTables': 145, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 11112448}, {u'TargetSize': 24708022272L, u'LogicalSize': 20739421856L, u'SizeRatio': 0.8393800858558657, u'NWriteBytes': 3046027264L, u'BloomFilterMemUsed': 0, u'NumTables': 947, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 205946880}, {u'TargetSize': 97030342756L, u'LogicalSize': 49416044544L, u'SizeRatio': 0.5092844479408406, u'NWriteBytes': 63798943744L, u'BloomFilterMemUsed': 0, u'NumTables': 161, u'WastedSpace': 900873166, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 49416044544L}], u'NItemsGarbageCollected': 19600917, u'NInternalFragmentationCompacts': 39, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 303360171, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 149760, u'NNonL0LocalCompacts': 10, u'NWriterCompacts': 0, u'NReadIOs': 10781381, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 439920, u'CompactWriter': {u'NDataBlockWrites': 3365047, u'NWriteBytes': 10147610624L, u'NIndexBlockWrites': 164992, u'NBytesIncoming': 13833643513L, u'NSyncs': 0, u'NInserts': 458723851, u'NWriteIOs': 157469}, u'NWriteBytes': 81046315336L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 885093, u'NReadBytes': 2827363599L, u'NDataBlockReads': 874371, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2623113}, u'NBloomFilterMisses': 0, u'NItemsMerged': 18391369, u'LogicalDiskSize': 49416044544L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.8596205152423946, u'BufferMemUsed': 362780, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 28676622688L, u'NTableFiles': 1520, u'NFileCountCompacts': 406, u'TableObjectMemUsed': 386880, u'TreeSnapshotMemoryUsed': 162752, u'TotalDiskUsage': 59260776448L, u'BytesIncoming': 94281504337L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 45003, u'NItemsPut': 99812291, u'NSyncs': 66161, u'CompactReader': {u'NReadIOs': 9896098, u'NReadBytes': 29545866374L, u'NDataBlockReads': 9639879, u'NBytesOutgoing': 0, u'NIndexBlockReads': 256219}, u'ActiveDiskUsage': 49634742272L, u'NTablesDeleted': 197940, u'IterateReader': {u'NReadIOs': 190, u'NReadBytes': 105265, u'NDataBlockReads': 95, u'NBytesOutgoing': 0, u'NIndexBlockReads': 95}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 21466873, u'NWriteBytes': 70898704712L, u'NIndexBlockWrites': 541027, u'NBytesIncoming': 93403813635L, u'NSyncs': 66161, u'NInserts': 98179841, u'NWriteIOs': 333672}}, u'NReadBytes': 63837332627277L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 109.53333282470703, u'NInternalFragmentationCompacts': 39, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 322713877, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 8192, u'NTxns': 9359964, u'NWriteBytes': 156717957120L, u'NSegments': 10120, u'NSyncs': 9359964, u'NLoggedBytes': 156717957120L, u'NActiveSegments': 3, u'NWrites': 9392751, u'NInserts': 165008242, u'DiskUsed': 34275328}, u'NGets': 9112653, u'NReadBytesSet': 67416451851L, u'NReadBytesCompact': 76435897613L, u'IndexResidentRatio': 0.06334228813648224, u'CompactQueueSize': 0, u'NSyncsPerSec': 0.800000011920929, u'BlockCacheHitRatio': 70.17050170898438, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 63760896624399L, u'BloomFilterMemUsedRatio': 0.18876691162586212, u'NWriteBytes': 289333813958L, u'BytesOutgoing': 1232637795, u'RecentReadIOAmp': 2.0990195274353027, u'WriteAmp': 2.9771995544433594, u'NBlocksCached': 28888, u'LogicalDataSize': 29832567831L, u'ActiveBloomFilterMemUsed': 31003048, u'NFileCountCompacts': 15296, u'NTableFiles': 2083, u'FlushQueueSize': 72, u'TableObjectMemUsed': 526504, u'BlockCacheQuota': 95940558, u'TotalDiskUsage': 60694544384L, u'NPurgerRuns': 203, u'ReadIOAmpSet': 0.0, u'NSyncs': 198483, u'NReadIOsSetPerSec': 55.86666488647461, u'NReadIOsPerSec': 217.6999969482422, u'NTablesDeleted': 478755, u'BloomFilterFPR': 0.015265412628650665, u'NBloomFilterHitsPerSec': 2200.86669921875, u'WriteCacheMemUsed': 12193472, u'RecentBloomFilterFPR': 0.019687611609697342, u'BytesPerRead': 6989565.0, u'BytesIncomingPerSec': 373055.03125, u'NReadIOsGetPerSec': 214.10000610351562, u'NGetStats': 332440, u'TotalIndexBlocksSize': 379981917, u'NGetStatsPerSec': 1.6333333253860474, u'NWriteBytesCompact': 55417774080L, u'NReadIOsSet': 25296290, u'ActiveDataSize': 29832567831L, u'NWriteBytesCompactPerSec': 819.2000122070312, u'NBloomFilterMissesPerSec': 44.20000076293945, u'NCommitBatches': 7184816, u'NGetStatsComputed': 332440, u'NWriteIOs': 10376817, u'NReadIOsGet': 23895863500L, u'RecentReadAmpGet': 6.233266353607178, u'WriteCacheQuota': 23985140, u'LSMTreeObjectMemUsed': 132608, u'NSetsPerSec': 281.3999938964844, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 0.5333333611488342, u'ManagedFileObjectMemUsed': 203808, u'ReadAmpGet': 51672.5, u'BlockCacheMemUsed': 96042400, u'NWriterCompacts': 0, u'NReadIOs': 23923453078L, u'WALBufferMemUsed': 65536, u'ReadAmp': 51734.51171875, u'keyStats': {u'NItemsTransformed': 1081231776, u'NTablesCreated': 147581, u'ReadIOAmp': 317.5595609724027, u'MemtablesMemUsed': 711232, u'NBloomFilterHits': 674815286, u'TotalIndexBlocksSize': 19582680, u'CheckpointOverhead': 18432000, u'NItemsGet': 75245657, u'NWriteIOs': 293150, u'NReadBytes': 63804828321439L, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 2059368, u'SizeRatio': 0.06137394905090332, u'NWriteBytes': 4501831680L, u'BloomFilterMemUsed': 28312, u'NumTables': 69, u'WastedSpace': 24, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1957888}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 26798168, u'SizeRatio': 0.7986476421356201, u'NWriteBytes': 16800894976L, u'BloomFilterMemUsed': 892928, u'NumTables': 15, u'WastedSpace': 63400, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 26861568}, {u'TargetSize': 126812127, u'LogicalSize': 112267818, u'SizeRatio': 0.8853082166187466, u'NWriteBytes': 8728817664L, u'BloomFilterMemUsed': 3841024, u'NumTables': 56, u'WastedSpace': 300206, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 112566272}, {u'TargetSize': 1268121297, u'LogicalSize': 1268121297, u'SizeRatio': 1.0, u'NWriteBytes': 14485172224L, u'BloomFilterMemUsed': 26239648, u'NumTables': 177, u'WastedSpace': 4471539, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1272582144}], u'NItemsGarbageCollected': 31213639, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 19350682, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 38016, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 23912403988L, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 799300, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 139352, u'CompactWriter': {u'NDataBlockWrites': 16535460, u'NWriteBytes': 45209378816L, u'NIndexBlockWrites': 213566, u'NBytesIncoming': 63008007186L, u'NSyncs': 0, u'NInserts': 1089867466, u'NWriteIOs': 100588}, u'NWriteBytes': 50440992180L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 23894977802L, u'NReadBytes': 63758068979187L, u'NDataBlockReads': 42926425380L, u'NBytesOutgoing': 0, u'NIndexBlockReads': 84256911637L}, u'NBloomFilterMisses': 10461013, u'NItemsMerged': 0, u'LogicalDiskSize': 1268121297, u'TotalBloomFilterMemUsed': 28149560, u'WriteAmp': 15.541378328790033, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 31001912, u'LogicalDataSize': 1155853479, u'NTableFiles': 396, u'NFileCountCompacts': 7470, u'TableObjectMemUsed': 98208, u'TreeSnapshotMemoryUsed': 65856, u'TotalDiskUsage': 1432399872, u'BytesIncoming': 3245593223L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 9955, u'NItemsPut': 66132719, u'NSyncs': 66161, u'CompactReader': {u'NReadIOs': 17426186, u'NReadBytes': 46759342252L, u'NDataBlockReads': 17092216, u'NBytesOutgoing': 0, u'NIndexBlockReads': 333970}, u'ActiveDiskUsage': 1413967872, u'NTablesDeleted': 147263, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.015265394433260561, u'Writer': {u'NDataBlockWrites': 1081720, u'NWriteBytes': 5231613364L, u'NIndexBlockWrites': 126690, u'NBytesIncoming': 3770185622L, u'NSyncs': 66161, u'NInserts': 64915625, u'NWriteIOs': 192562}}, u'NReadBytesCompactPerSec': 1745.2332763671875, u'TableMetaMemUsed': 627226, u'NDeletes': 0, u'BlockCacheHitsPerSec': 706.2333374023438, u'BytesOutgoingPerSec': 96264.5703125, u'NBloomFilterMisses': 10461029, u'LogicalDiskSize': 50684257505L, u'NSets': 66132719, u'TotalBloomFilterMemUsed': 28150960, u'BufferMemUsed': 362780, u'RecentReadAmp': 6.251396179199219, u'NGetsPerSec': 102.0, u'NInserts': 0, u'NCommitBatchesPerSec': 180.1999969482422, u'FragmentationWithOverhead': 0.5084802508354187, u'NCompactsPerSec': 0.10000000149011612, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 601787.9375, u'TreeSnapshotMemoryUsed': 261632, u'BytesIncoming': 97183208566L, u'NCompacts': 62378, u'NumPersistentFlushes': 63954, u'NWriteIOsPerSec': 183.23333740234375, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 5882.771484375, u'ActiveDiskUsage': 51049209856L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 1.6333333253860474, u'NReadBytesSetPerSec': 146443.40625, u'BlockCacheHits': 142691895, u'NWriteBytesPerSec': 914922.8125, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 133614, u'ReadIOAmp': 0.02967431822640769, u'MemtablesMemUsed': 3316288, u'NBloomFilterHits': 223, u'TotalIndexBlocksSize': 8208, u'CheckpointOverhead': 720896, u'NItemsGet': 20388, u'NWriteIOs': 199775, u'NReadBytes': 130970600, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 491672, u'SizeRatio': 0.01465296745300293, u'NWriteBytes': 907988992, u'BloomFilterMemUsed': 624, u'NumTables': 53, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 434176}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 91664, u'SizeRatio': 0.002731800079345703, u'NWriteBytes': 53387264, u'BloomFilterMemUsed': 512, u'NumTables': 8, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 65536}], u'NItemsGarbageCollected': 34884681, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 3024, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 16032, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 267709, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 47954, u'CompactWriter': {u'NDataBlockWrites': 7420, u'NWriteBytes': 60784640, u'NIndexBlockWrites': 7420, u'NBytesIncoming': 33676391, u'NSyncs': 0, u'NInserts': 393260, u'NWriteIOs': 7420}, u'NWriteBytes': 1128549322, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 605, u'NReadBytes': 281613, u'NDataBlockReads': 553, u'NBytesOutgoing': 0, u'NIndexBlockReads': 553}, u'NBloomFilterMisses': 16, u'NItemsMerged': 0, u'LogicalDiskSize': 91664, u'TotalBloomFilterMemUsed': 1400, u'WriteAmp': 0.27304947314675104, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1136, u'LogicalDataSize': 91664, u'NTableFiles': 167, u'NFileCountCompacts': 7420, u'TableObjectMemUsed': 41416, u'TreeSnapshotMemoryUsed': 33024, u'TotalDiskUsage': 1368064, u'BytesIncoming': 4133131293L, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 7420, u'NItemsPut': 34896266, u'NSyncs': 66161, u'CompactReader': {u'NReadIOs': 267104, u'NReadBytes': 130688987, u'NDataBlockReads': 133552, u'NBytesOutgoing': 0, u'NIndexBlockReads': 133552}, u'ActiveDiskUsage': 499712, u'NTablesDeleted': 133552, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.06694560669456066, u'Writer': {u'NDataBlockWrites': 126194, u'NWriteBytes': 1067764682, u'NIndexBlockWrites': 126194, u'NBytesIncoming': 92444981, u'NSyncs': 66161, u'NInserts': 1215009, u'NWriteIOs': 192355}}}}} 2021-12-15 15:51:39,957 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.68': [0.027961039915680885, 0.02857033908367157, 0.028953777626156807, 0.028315942734479904, 0.028682677075266838, 0.028691468760371208, 0.029477065429091454, 0.029870759695768356, 0.03031134605407715, 0.028095021843910217, 0.02802875079214573, 0.02820712700486183, 0.029143154621124268, 0.0288242157548666, 0.026691418141126633, 0.027822265401482582, 0.024765659123659134, 0.024686120450496674, 0.02437589317560196, 0.025035878643393517, 0.025654690340161324, 0.02630523592233658, 0.024278895929455757, 0.024523066356778145, 0.025040652602910995, 0.026103559881448746, 0.02561591938138008, 0.0225082878023386, 0.023260049521923065, 0.02360359951853752, 0.024103466421365738, 0.024865293875336647, 0.0254664309322834, 0.02648918516933918, 0.026020826771855354, 0.022812796756625175, 0.023576926440000534, 0.024250173941254616, 0.024871665984392166, 0.025037666782736778, 0.0228243637830019, 0.022733932361006737, 0.02358982153236866, 0.024637453258037567, 0.02502593770623207, 0.025469835847616196, 0.02604784071445465, 0.025957081466913223, 0.02458874322474003, 0.02463850937783718, 0.025276536121964455, 0.02557552233338356, 0.026162927970290184, 0.026653120294213295, 0.02333028055727482, 0.023161591961979866, 0.023956621065735817, 0.024513766169548035, 0.025015434250235558, 0.025294268503785133, 0.025422751903533936, 0.02498684637248516, 0.027251187711954117, 0.02740098349750042], '172.23.110.65': [0.41140368580818176, 0.39665400981903076, 0.4241296648979187, 0.4013958275318146, 0.4185928702354431, 0.4298474192619324, 0.43011704087257385, 0.4044402539730072, 0.440847247838974, 0.43156808614730835, 0.4367906451225281, 0.45116493105888367, 0.44563615322113037, 0.41989922523498535, 0.42198967933654785, 0.43946200609207153, 0.43713030219078064, 0.42008164525032043, 0.4558884799480438, 0.4139108955860138, 0.4212176203727722, 0.4277351200580597, 0.45784246921539307, 0.44580623507499695, 0.417273610830307, 0.42464306950569153, 0.4418584704399109, 0.4552067518234253, 0.44015321135520935, 0.4460470378398895, 0.431092768907547, 0.40214410424232483, 0.43872329592704773, 0.42464375495910645, 0.39889228343963623, 0.42678302526474, 0.4484919011592865, 0.44598326086997986, 0.3958740532398224, 0.41754552721977234, 0.3857172131538391, 0.4670185148715973, 0.43315058946609497, 0.452467679977417, 0.4317549765110016, 0.3880397081375122, 0.44735318422317505, 0.3864130675792694, 0.3895998001098633, 0.42973169684410095, 0.4046151340007782, 0.4312382936477661, 0.40386807918548584, 0.43009254336357117, 0.4059758186340332, 0.4239043593406677, 0.41711050271987915, 0.43080392479896545, 0.4039328098297119, 0.42857566475868225, 0.4480356276035309, 0.39513394236564636, 0.4384009838104248, 0.4194563329219818], '172.23.110.66': [0.42626845836639404, 0.43545153737068176, 0.37815067172050476, 0.4445668160915375, 0.3925720453262329, 0.43095794320106506, 0.42337653040885925, 0.4465816020965576, 0.43354904651641846, 0.39509886503219604, 0.41324955224990845, 0.41411063075065613, 0.41518524289131165, 0.42332562804222107, 0.4266292452812195, 0.4412384331226349, 0.4100286662578583, 0.4500778913497925, 0.4306544065475464, 0.4250417947769165, 0.3710751235485077, 0.4213303029537201, 0.400285005569458, 0.4190075099468231, 0.4361043870449066, 0.41563907265663147, 0.427722692489624, 0.40961045026779175, 0.43740880489349365, 0.4335455000400543, 0.44387757778167725, 0.4126097559928894, 0.42639869451522827, 0.42702895402908325, 0.44279593229293823, 0.41407889127731323, 0.4122360348701477, 0.4410210847854614, 0.43686485290527344, 0.44055142998695374, 0.4499564468860626, 0.4462072551250458, 0.42258667945861816, 0.4328564703464508, 0.38876578211784363, 0.4387814700603485, 0.4069886803627014, 0.4140198528766632, 0.43458253145217896, 0.44344276189804077, 0.4410705268383026, 0.4132424592971802, 0.4381379187107086, 0.4343125522136688, 0.4081498384475708, 0.44127821922302246, 0.3998119533061981, 0.42360639572143555, 0.4074423313140869, 0.44413256645202637, 0.4613288640975952, 0.42366746068000793, 0.4274837076663971, 0.39371457695961], '172.23.110.64': [0.4348811209201813, 0.44702136516571045, 0.42494896054267883, 0.44255414605140686, 0.4634062647819519, 0.420172780752182, 0.44766780734062195, 0.41982102394104004, 0.43608182668685913, 0.44234374165534973, 0.42440125346183777, 0.4270927608013153, 0.4312402307987213, 0.38520729541778564, 0.4382932782173157, 0.387762188911438, 0.39342761039733887, 0.41717982292175293, 0.43748605251312256, 0.421178936958313, 0.43405941128730774, 0.4397357404232025, 0.41695818305015564, 0.37931257486343384, 0.4671124517917633, 0.42867571115493774, 0.42890381813049316, 0.44575002789497375, 0.40254563093185425, 0.38723382353782654, 0.46602174639701843, 0.419629842042923, 0.43024763464927673, 0.440000981092453, 0.3986052870750427, 0.41973942518234253, 0.42786046862602234, 0.40050825476646423, 0.4407047629356384, 0.4316684603691101, 0.3832322359085083, 0.44448134303092957, 0.4463815987110138, 0.4302673935890198, 0.4264923930168152, 0.4014023542404175, 0.4029453992843628, 0.43528619408607483, 0.440399169921875, 0.433643102645874, 0.421497642993927, 0.4409833550453186, 0.41422730684280396, 0.4328649938106537, 0.4144533574581146, 0.42073142528533936, 0.4348224103450775, 0.403103768825531, 0.40631306171417236, 0.41922783851623535, 0.41504013538360596, 0.38810834288597107, 0.43648362159729004, 0.4143387973308563]} 2021-12-15 15:51:40,217 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 35.0 2021-12-15 15:51:40,558 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 35.0 2021-12-15 15:51:40,773 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.68 is 35.0 2021-12-15 15:51:40,994 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 35.0 2021-12-15 15:51:40,994 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.68': 35.0, '172.23.110.65': 35.0, '172.23.110.66': 35.0, '172.23.110.64': 35.0} ############################################################ # # Step 5: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 5426.4453125 # ############################################################ 2021-12-15 15:51:41,052 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 15:51:41,052 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 102 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-15 15:53:23,058 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 15:53:23,059 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 15:53:28,062 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:53:28,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:28,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:53:28,556 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:28,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:53:28,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:53:28,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:53:28,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:53:28,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:53:28,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:53:28,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:53:28,651 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:53:28,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:53:28,710 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['297000 memcached\n'], []) 2021-12-15 15:53:28,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 15:53:28,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:53:28,796 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['297000 memcached\n'], []) 2021-12-15 15:53:28,796 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:53:28,802 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:53:28,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:29,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:53:29,211 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:29,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:53:29,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:53:29,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:53:29,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:53:29,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:53:29,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:53:29,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:53:29,335 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:53:29,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:53:29,423 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['265469 memcached\n'], []) 2021-12-15 15:53:29,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 15:53:29,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:53:29,535 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['265469 memcached\n'], []) 2021-12-15 15:53:29,536 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:53:29,539 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:53:29,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:29,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:53:30,022 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:30,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:53:30,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:53:30,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:53:30,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:53:30,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:53:30,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:53:30,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:53:30,144 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:53:30,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:53:30,194 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['32634 memcached\n'], []) 2021-12-15 15:53:30,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 15:53:30,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:53:30,325 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['32634 memcached\n'], []) 2021-12-15 15:53:30,325 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:53:30,328 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:53:30,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:30,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:53:30,802 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:30,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:53:30,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:53:30,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:53:30,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:53:30,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:53:30,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:53:30,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:53:30,938 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:53:30,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:53:31,015 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['123526 memcached\n'], []) 2021-12-15 15:53:31,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 15:53:31,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:53:31,163 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['123526 memcached\n'], []) 2021-12-15 15:53:31,164 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:53:31,167 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:53:31,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:31,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:53:31,569 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:31,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:53:31,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:53:31,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:53:31,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:53:31,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:53:31,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:53:31,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:53:31,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:53:31,703 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 15:53:31,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:53:31,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:31,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:31,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:31,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:31,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:31,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:32,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:32,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:32,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:32,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:32,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:32,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:32,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:32,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:32,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:53:32,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:53:32,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:32,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:32,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:53:32,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:53:32,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:32,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:32,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:53:32,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:53:32,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:53:32,710 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 15:53:32,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:53:32,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:32,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:53:32,775 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 15:53:32,776 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:53:32,779 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:53:32,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:33,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:53:33,288 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:33,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:53:33,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:53:33,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:53:33,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:53:33,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:53:33,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:53:33,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:53:33,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:53:33,434 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 15:53:33,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:53:33,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:33,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:33,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:33,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:33,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:33,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:33,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:33,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:33,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:33,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:33,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:33,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:33,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:53:33,964 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 15:53:33,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:53:33,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:34,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:53:34,023 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 15:53:34,023 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:53:34,029 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:53:34,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:34,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:53:34,519 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:34,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:53:34,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:53:34,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:53:34,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:53:34,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:53:34,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:53:34,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:53:34,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:53:34,660 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 15:53:34,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:53:34,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:34,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:34,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:34,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:34,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:53:34,868 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 15:53:34,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:53:34,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:34,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:53:34,927 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 15:53:34,928 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:53:34,931 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:53:35,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:35,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:53:35,469 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:35,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:53:35,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:53:35,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:53:35,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:53:35,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:53:35,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:53:35,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:53:35,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:53:35,601 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 15:53:35,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:53:35,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:35,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:35,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:35,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:53:35,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:35,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:35,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:35,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:53:35,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:35,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:53:35,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:35,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:53:35,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:53:35,970 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 15:53:35,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:53:35,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:53:36,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:53:36,013 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 15:53:36,013 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:53:36,013 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:53:36,016 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:53:36,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:53:36,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:53:36,473 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:53:36,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:53:36,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:53:36,512 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:53:36,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:53:36,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:53:36,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:53:36,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:53:36,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:40,410 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 15:53:40,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:40,418 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:42,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:42,596 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:42,602 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:42,602 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:44,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:44,783 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:44,789 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:44,789 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:46,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:46,946 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:46,951 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:46,951 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:48,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:49,128 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:49,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:49,135 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:51,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:51,260 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:51,266 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:51,266 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:53,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:53,444 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:53,448 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:53,450 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:55,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:55,624 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:55,628 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:55,628 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:57,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:57,778 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:57,782 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:57,783 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:53:59,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:53:59,934 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:53:59,940 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:53:59,940 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:01,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:02,085 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:54:02,092 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:02,092 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:04,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:04,276 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:54:04,280 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:04,280 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:06,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:06,447 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:54:06,453 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:06,453 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:08,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:08,584 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:54:08,588 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:08,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:10,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:10,727 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:10,729 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:12,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:12,957 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:12,957 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:14,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:15,181 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:15,181 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:17,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:17,355 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:17,355 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:19,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:19,525 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:19,525 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:21,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:21,734 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:21,736 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:23,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:23,964 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:23,966 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:25,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:26,153 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:26,153 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:28,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:28,382 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:28,382 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:30,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:30,608 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:30,608 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:32,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:32,801 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:32,801 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:34,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:34,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:34,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:36,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:37,196 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:37,196 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:39,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:39,400 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:39,400 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:41,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:41,576 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:41,576 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:43,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:43,727 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:43,727 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:45,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:45,921 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:45,921 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:47,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:48,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:48,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:50,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:50,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:50,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:52,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:52,585 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:52,585 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:54,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:54,801 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:54,801 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:56,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:57,609 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:57,609 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:54:59,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:54:59,835 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:54:59,835 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:01,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:02,017 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:02,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:04,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:04,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:04,198 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:06,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:06,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:06,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:08,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:08,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:08,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:10,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:10,834 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:10,834 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:55:12,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:13,161 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:55:13,163 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 97.1490001678. 2021-12-15 15:55:13,164 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:55:13,165 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:55:13,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:55:13,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:55:13,766 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:55:13,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:55:13,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:55:13,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:55:13,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:55:13,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:55:13,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:55:13,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:55:13,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:14,124 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:55:14,124 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.959000110626. 2021-12-15 15:55:14,124 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:55:14,127 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:55:14,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:55:14,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:55:14,638 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:55:14,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:55:14,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:55:14,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:55:14,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:55:14,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:55:14,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:55:14,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:55:14,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:14,930 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:55:14,930 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.805999994278. 2021-12-15 15:55:14,931 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:55:14,936 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:55:15,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:55:15,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:55:15,351 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:55:15,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:55:15,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:55:15,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:55:15,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:55:15,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:55:15,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:55:15,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:55:15,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:15,650 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:15,651 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 15:55:17,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:17,809 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:17,809 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 15:55:19,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:19,971 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:55:19,971 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 15:55:21,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:55:22,201 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:55:22,203 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.26999998093. ############################################################ # # RAM FootPrint: 5567.5703125 # ############################################################ 2021-12-15 15:55:22,236 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 15:55:22,236 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-15 15:56:50,243 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 15:56:50,243 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 15:56:55,244 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:56:55,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:56:55,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:56:55,882 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:56:55,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:56:55,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:56:55,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:56:55,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:56:56,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:56:56,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:56:56,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:56:56,063 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:56:56,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:56:56,144 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['82249 memcached\n'], []) 2021-12-15 15:56:56,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 15:56:56,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:56:56,319 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['82249 memcached\n'], []) 2021-12-15 15:56:56,319 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:56:56,322 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:56:56,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:56:56,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:56:56,921 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:56:56,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:56:56,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:56:56,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:56:56,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:56:56,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:56:57,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:56:57,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:56:57,059 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:56:57,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:56:57,108 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['50510 memcached\n'], []) 2021-12-15 15:56:57,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 15:56:57,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:56:57,267 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['50510 memcached\n'], []) 2021-12-15 15:56:57,267 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:56:57,270 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:56:57,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:56:57,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:56:57,776 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:56:57,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:56:57,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:56:57,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:56:57,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:56:57,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:56:57,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:56:57,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:56:57,927 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:56:57,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:56:58,009 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['100005 memcached\n'], []) 2021-12-15 15:56:58,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 15:56:58,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:56:58,142 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['100005 memcached\n'], []) 2021-12-15 15:56:58,144 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:56:58,148 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:56:58,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:56:58,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:56:58,687 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:56:58,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:56:58,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:56:58,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:56:58,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:56:58,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:56:58,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:56:58,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:56:58,838 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:56:58,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:56:58,933 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['127870 memcached\n'], []) 2021-12-15 15:56:58,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 15:56:59,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:56:59,095 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['127870 memcached\n'], []) 2021-12-15 15:56:59,095 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:56:59,098 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:56:59,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:56:59,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:56:59,509 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:56:59,510 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:56:59,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:56:59,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:56:59,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:56:59,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:56:59,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:56:59,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:56:59,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:56:59,674 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 15:56:59,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:56:59,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:56:59,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:56:59,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:56:59,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:56:59,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:56:59,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:56:59,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:00,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:00,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:00,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:00,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:00,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:00,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:00,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:00,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:57:00,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:57:00,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:00,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:00,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:57:00,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:57:00,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:00,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:00,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:57:00,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:57:00,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:57:00,492 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 15:57:00,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:57:00,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:00,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:57:00,529 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 15:57:00,529 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:57:00,532 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:57:00,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:57:00,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:57:01,038 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:57:01,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:57:01,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:57:01,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:57:01,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:57:01,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:57:01,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:57:01,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:57:01,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:57:01,188 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 15:57:01,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:57:01,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:01,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:01,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:01,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:01,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:01,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:01,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:01,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:01,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:01,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:01,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:01,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:01,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:01,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:01,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:57:01,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:57:01,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:57:01,845 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 15:57:01,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:57:01,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:01,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:57:01,895 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 15:57:01,895 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:57:01,898 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:57:02,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:57:02,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:57:02,374 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:57:02,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:57:02,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:57:02,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:57:02,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:57:02,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:57:02,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:57:02,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:57:02,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:57:02,517 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 15:57:02,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:57:02,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:02,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:02,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:02,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:02,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:02,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:02,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:02,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:02,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:57:02,802 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 15:57:02,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:57:02,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:02,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:57:02,848 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 15:57:02,848 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:57:02,851 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:57:02,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:57:03,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:57:03,392 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:57:03,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:57:03,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:57:03,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:57:03,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:57:03,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:57:03,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:57:03,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:57:03,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:57:03,535 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 15:57:03,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:57:03,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:03,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:03,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:03,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:57:03,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:03,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:03,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:03,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:57:03,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:03,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:57:03,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:03,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:57:03,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:57:03,990 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 15:57:03,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:57:04,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:57:04,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:57:04,030 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 15:57:04,032 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:57:04,032 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:57:04,035 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:57:04,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:57:04,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:57:04,517 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:57:04,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:57:04,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:57:04,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:57:04,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:57:04,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:57:04,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:57:04,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:57:04,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:04,848 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 15:57:04,854 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:04,854 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:06,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:07,016 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:07,022 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:07,023 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:09,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:09,203 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:09,209 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:09,209 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:11,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:11,334 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:11,338 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:11,338 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:13,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:13,516 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:13,523 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:13,523 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:15,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:15,700 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:15,704 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:15,704 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:17,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:17,888 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:17,895 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:17,895 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:19,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:20,075 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:20,081 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:20,081 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:22,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:22,266 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:22,272 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:22,272 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:24,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:24,457 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 15:57:24,460 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:24,461 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:26,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:26,687 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:26,687 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:28,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:28,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:28,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:30,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:31,082 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:31,082 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:33,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:33,257 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:33,257 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:35,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:35,444 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:35,444 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:37,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:37,681 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:37,683 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:39,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:39,894 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:39,894 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:41,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:42,101 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:42,101 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:44,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:44,256 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:44,256 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:46,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:46,447 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:46,447 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:48,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:48,599 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:48,601 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:50,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:50,790 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:50,790 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:52,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:52,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:52,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:54,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:55,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:55,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:57,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:57,385 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:57,385 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:57:59,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:57:59,612 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:57:59,612 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:01,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:01,848 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:01,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:03,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:04,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:04,055 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:06,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:06,234 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:06,236 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:08,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:08,446 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:08,446 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:10,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:10,667 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:10,668 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:12,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:12,808 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:12,808 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:14,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:15,023 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:15,023 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:17,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:17,200 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:17,200 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:19,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:19,415 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:19,417 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:21,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:21,586 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:21,586 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:23,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:23,756 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:23,756 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:25,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:25,986 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:25,986 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:27,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:28,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 15:58:28,173 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 15:58:30,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:32,483 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:58:32,483 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 88.4509999752. 2021-12-15 15:58:32,484 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:58:32,486 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:58:32,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:58:32,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:58:33,131 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:58:33,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:58:33,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:58:33,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:58:33,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:58:33,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:58:33,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:58:33,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:58:33,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:33,480 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:58:33,482 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.997000217438. 2021-12-15 15:58:33,482 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:58:33,484 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:58:33,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:58:33,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:58:34,005 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:58:34,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:58:34,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:58:34,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:58:34,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:58:34,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:58:34,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:58:34,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:58:34,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:34,282 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:58:34,282 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.800999879837. 2021-12-15 15:58:34,285 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 15:58:34,286 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:58:34,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:58:34,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:58:34,710 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:58:34,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:58:34,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:58:34,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:58:34,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:58:34,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:58:39,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:58:39,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:58:39,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 15:58:39,493 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:58:39,494 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.20999979973. ############################################################ # # RAM FootPrint: 5569.6484375 # ############################################################ 2021-12-15 15:58:39,530 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 15:58:39,532 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-15 15:59:47,536 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 15:59:47,536 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 15:59:52,538 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:59:52,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:52,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:59:53,075 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:53,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:59:53,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:59:53,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:59:53,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:59:53,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:59:53,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:59:53,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:59:53,211 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:59:53,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:59:53,279 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['83621 memcached\n'], []) 2021-12-15 15:59:53,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 15:59:53,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 15:59:53,463 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['83621 memcached\n'], []) 2021-12-15 15:59:53,463 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:59:53,466 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 15:59:53,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:53,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 15:59:54,115 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:54,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 15:59:54,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 15:59:54,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 15:59:54,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 15:59:54,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 15:59:54,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 15:59:54,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 15:59:54,244 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:59:54,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:59:54,316 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['51358 memcached\n'], []) 2021-12-15 15:59:54,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 15:59:55,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 15:59:55,368 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['51358 memcached\n'], []) 2021-12-15 15:59:55,368 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 15:59:55,371 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 15:59:55,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:55,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 15:59:55,858 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:55,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 15:59:55,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 15:59:55,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 15:59:55,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 15:59:55,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 15:59:55,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 15:59:55,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 15:59:55,986 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:59:55,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:59:56,042 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['100793 memcached\n'], []) 2021-12-15 15:59:56,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 15:59:56,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 15:59:56,168 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['100793 memcached\n'], []) 2021-12-15 15:59:56,168 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 15:59:56,171 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:59:56,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:56,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:59:56,686 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:56,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:59:56,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:59:56,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:59:56,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:59:56,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:59:56,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:59:56,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:59:56,898 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 15:59:56,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:59:56,973 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['128974 memcached\n'], []) 2021-12-15 15:59:56,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 15:59:57,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 15:59:57,112 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['128974 memcached\n'], []) 2021-12-15 15:59:57,112 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:59:57,115 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 15:59:57,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:57,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 15:59:57,579 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:57,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 15:59:57,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 15:59:57,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 15:59:57,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 15:59:57,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 15:59:57,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 15:59:57,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 15:59:57,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:59:57,729 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 15:59:57,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:59:57,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:57,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:57,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:59:57,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:59:57,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:57,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:58,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:59:58,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:59:58,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:58,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:58,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:59:58,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:59:58,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:58,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:58,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:59:58,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:59:58,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:58,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:58,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:59:58,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 15:59:58,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:58,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:58,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:59:58,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 15:59:58,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:59:58,796 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 15:59:58,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:59:58,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:58,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:59:58,862 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 15:59:58,864 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 15:59:58,867 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 15:59:58,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 15:59:59,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 15:59:59,362 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 15:59:59,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 15:59:59,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 15:59:59,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 15:59:59,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 15:59:59,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 15:59:59,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 15:59:59,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 15:59:59,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 15:59:59,502 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 15:59:59,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 15:59:59,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:59,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:59,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:59:59,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 15:59:59,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:59,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:59,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:59:59,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 15:59:59,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:59,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:59,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:59:59,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 15:59:59,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:59,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 15:59:59,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:59:59,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 15:59:59,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 15:59:59,943 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 15:59:59,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 15:59:59,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 15:59:59,971 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 15:59:59,986 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 15:59:59,986 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 15:59:59,990 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:00:00,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:00:00,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:00:00,476 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:00:00,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:00:00,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:00:00,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:00:00,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:00:00,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:00:00,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:00:00,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:00:00,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:00:00,628 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:00:00,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:00:00,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:00,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:00:00,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:00:00,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:00:00,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:00,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:00:00,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:00:00,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:00:00,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:00:00,898 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:00:00,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:00:00,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:00,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:00:00,943 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:00:00,943 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:00:00,946 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:00:01,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:00:01,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:00:01,483 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:00:01,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:00:01,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:00:01,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:00:01,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:00:01,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:00:01,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:00:01,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:00:01,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:00:01,618 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:00:01,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:00:01,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:01,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:00:01,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:00:01,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:00:01,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:01,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:00:01,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:00:01,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:00:01,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:01,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:00:02,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:00:02,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:00:02,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:00:02,055 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:00:02,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:00:02,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:00:02,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:00:02,094 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:00:02,094 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:00:02,094 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:00:02,096 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:00:02,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:00:02,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:00:02,617 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:00:02,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:00:02,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:00:02,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:00:02,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:00:02,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:00:02,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:00:02,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:00:02,737 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:02,891 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: 'ep_warmup_thread' 2021-12-15 16:00:02,898 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:02,898 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:04,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:05,036 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:05,052 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:05,052 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:07,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:07,168 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:07,174 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:07,174 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:09,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:09,354 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:09,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:09,358 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:11,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:11,542 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:11,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:11,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:13,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:13,724 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:13,729 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:13,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:15,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:15,907 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:15,913 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:15,913 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:17,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:18,092 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:00:18,098 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:18,098 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:20,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:20,244 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:20,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:22,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:22,384 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:22,384 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:24,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:24,566 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:24,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:26,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:26,780 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:26,780 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:28,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:29,015 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:29,016 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:31,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:31,246 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:31,246 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:33,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:33,469 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:33,469 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:35,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:35,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:35,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:37,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:37,882 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:37,884 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:39,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:40,068 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:40,068 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:42,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:42,242 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:42,243 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:44,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:44,427 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:44,427 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:46,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:46,660 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:46,660 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:48,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:48,834 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:48,834 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:50,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:51,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:51,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:53,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:53,266 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:53,266 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:55,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:55,499 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:55,499 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:57,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:57,698 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:57,700 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:00:59,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:00:59,957 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:00:59,957 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:01,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:02,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:02,153 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:04,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:04,394 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:04,394 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:06,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:06,575 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:06,575 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:08,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:08,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:08,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:10,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:10,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:10,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:12,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:13,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:13,194 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:15,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:15,368 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:15,368 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:17,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:17,539 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:17,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:19,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:19,700 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:19,700 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:21,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:21,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:01:21,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:01:23,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:24,117 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:01:24,117 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 82.0230000019. 2021-12-15 16:01:24,118 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:01:24,119 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:01:24,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:01:24,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:01:24,759 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:01:24,760 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:01:24,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:01:24,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:01:24,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:01:24,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:01:24,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:01:24,928 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:01:24,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:25,177 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:01:25,178 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.06199979782. 2021-12-15 16:01:25,178 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:01:25,181 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:01:25,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:01:25,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:01:25,700 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:01:25,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:01:25,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:01:25,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:01:25,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:01:25,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:01:25,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:01:25,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:01:25,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:26,036 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:01:26,036 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.858000040054. 2021-12-15 16:01:26,038 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:01:26,039 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:01:26,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:01:28,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:01:28,864 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:01:28,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:01:28,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:01:28,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:01:28,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:01:28,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:01:28,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:01:28,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:01:29,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:01:29,188 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:01:29,190 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.15199995041. ############################################################ # # RAM FootPrint: 5568.16015625 # ############################################################ 2021-12-15 16:01:29,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:01:29,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-15 16:02:35,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:02:35,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:02:40,243 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:02:40,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:44,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:02:44,599 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:44,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:02:44,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:02:44,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:02:44,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:02:44,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:02:44,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:02:44,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:02:44,739 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:02:44,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:02:44,802 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['84907 memcached\n'], []) 2021-12-15 16:02:44,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:02:44,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:02:44,937 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['84907 memcached\n'], []) 2021-12-15 16:02:44,937 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:02:44,940 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:02:45,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:45,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:02:45,405 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:45,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:02:45,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:02:45,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:02:45,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:02:45,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:02:45,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:02:45,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:02:45,546 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:02:45,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:02:45,601 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['52144 memcached\n'], []) 2021-12-15 16:02:45,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:02:45,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:02:45,736 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['52144 memcached\n'], []) 2021-12-15 16:02:45,736 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:02:45,740 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:02:45,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:46,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:02:46,249 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:46,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:02:46,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:02:46,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:02:46,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:02:46,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:02:46,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:02:46,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:02:46,394 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:02:46,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:02:46,466 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['101572 memcached\n'], []) 2021-12-15 16:02:46,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:02:46,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:02:46,592 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['101572 memcached\n'], []) 2021-12-15 16:02:46,592 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:02:46,595 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:02:46,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:46,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:02:47,071 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:47,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:02:47,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:02:47,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:02:47,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:02:47,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:02:47,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:02:47,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:02:47,520 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:02:47,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:02:47,573 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['129983 memcached\n'], []) 2021-12-15 16:02:47,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:02:47,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:02:47,729 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['129983 memcached\n'], []) 2021-12-15 16:02:47,729 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:02:47,733 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:02:47,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:48,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:02:48,220 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:48,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:02:48,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:02:48,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:02:48,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:02:48,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:02:48,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:02:48,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:02:48,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:02:48,365 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:02:48,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:02:48,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:48,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:48,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:48,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:48,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:48,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:48,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:48,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:48,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:48,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:48,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:48,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:48,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:48,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:48,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:02:48,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:02:49,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:49,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:49,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:02:49,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:02:49,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:49,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:49,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:02:49,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:02:49,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:49,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:49,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:02:49,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:02:49,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:02:49,440 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:02:49,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:02:49,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:49,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:02:49,493 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:02:49,493 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:02:49,496 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:02:49,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:49,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:02:50,042 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:50,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:02:50,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:02:50,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:02:50,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:02:50,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:02:50,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:02:50,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:02:50,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:02:50,184 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:02:50,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:02:50,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:50,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:50,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:50,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:50,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:50,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:50,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:50,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:50,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:50,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:50,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:50,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:50,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:50,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:50,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:02:50,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:02:50,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:02:50,835 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:02:50,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:02:50,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:50,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:02:50,885 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:02:50,885 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:02:50,888 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:02:51,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:51,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:02:51,384 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:51,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:02:51,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:02:51,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:02:51,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:02:51,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:02:51,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:02:51,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:02:51,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:02:51,522 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:02:51,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:02:51,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:51,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:51,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:51,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:51,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:51,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:51,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:51,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:51,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:02:51,792 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:02:51,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:02:51,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:51,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:02:51,839 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:02:51,839 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:02:51,842 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:02:51,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:52,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:02:52,362 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:52,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:02:52,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:02:52,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:02:52,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:02:52,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:02:52,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:02:52,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:02:52,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:02:52,503 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:02:52,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:02:52,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:52,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:52,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:52,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:02:52,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:52,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:52,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:52,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:02:52,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:52,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:02:52,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:52,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:02:52,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:02:53,013 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:02:53,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:02:53,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:02:53,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:02:53,062 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:02:53,062 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:02:53,062 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:02:53,065 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:02:53,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:02:53,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:02:54,075 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:02:54,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:02:54,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:02:54,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:02:54,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:02:54,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:02:54,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:02:54,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:02:54,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:02:54,380 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:02:54,387 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:02:54,388 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:02:56,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:02:56,555 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:02:56,563 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:02:56,563 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:02:58,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:02:58,750 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:02:58,756 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:02:58,756 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:00,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:00,934 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:03:00,938 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:00,938 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:02,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:03,118 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:03:03,124 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:03,124 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:05,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:05,301 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:03:05,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:05,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:07,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:07,492 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:03:07,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:07,496 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:09,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:09,664 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:03:09,668 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:09,668 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:11,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:11,881 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:11,881 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:13,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:14,115 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:14,115 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:16,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:16,351 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:16,351 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:18,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:18,556 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:18,556 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:20,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:20,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:20,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:22,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:22,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:22,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:24,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:25,210 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:25,210 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:27,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:27,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:27,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:29,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:29,641 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:29,641 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:31,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:31,790 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:31,790 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:33,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:33,996 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:33,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:35,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:36,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:36,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:38,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:38,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:38,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:40,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:40,674 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:40,674 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:42,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:42,900 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:42,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:44,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:45,141 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:45,141 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:47,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:47,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:47,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:49,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:49,644 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:49,644 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:51,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:51,881 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:51,881 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:53,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:54,105 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:54,105 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:56,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:56,355 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:56,355 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:03:58,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:03:58,576 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:03:58,578 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:00,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:00,877 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:00,877 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:02,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:03,078 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:03,078 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:05,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:05,247 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:05,247 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:07,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:07,473 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:07,473 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:09,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:09,632 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:09,634 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:11,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:11,871 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:11,871 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:04:13,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:14,119 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:04:14,121 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 81.0580000877. 2021-12-15 16:04:14,121 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:04:14,125 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:04:15,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:04:15,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:04:15,908 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:04:15,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:04:15,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:04:15,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:04:15,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:04:16,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:04:16,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:04:16,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:04:16,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:16,252 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:04:16,253 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.13199996948. 2021-12-15 16:04:16,255 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:04:16,256 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:04:16,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:04:16,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:04:16,756 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:04:16,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:04:16,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:04:16,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:04:16,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:04:16,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:04:16,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:04:16,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:04:16,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:17,104 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:04:17,105 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.849999904633. 2021-12-15 16:04:17,105 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:04:17,108 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:04:17,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:04:17,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:04:17,446 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:04:17,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:04:17,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:04:17,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:04:17,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:04:17,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:04:17,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:04:17,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:04:17,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:17,749 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:04:17,749 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:04:19,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:04:19,984 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:04:19,984 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.8789999485. ############################################################ # # RAM FootPrint: 5599.45703125 # ############################################################ 2021-12-15 16:04:20,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:04:20,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-15 16:05:54,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:05:54,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:05:59,028 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:05:59,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:03,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:06:03,631 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:03,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:06:03,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:06:03,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:06:03,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:06:03,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:06:03,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:06:03,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:06:03,749 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:06:03,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:06:03,799 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['86125 memcached\n'], []) 2021-12-15 16:06:03,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:06:03,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:06:03,917 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['86125 memcached\n'], []) 2021-12-15 16:06:03,917 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:06:03,921 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:06:04,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:04,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:06:04,565 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:04,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:06:04,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:06:04,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:06:04,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:06:04,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:06:04,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:06:04,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:06:04,772 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:06:04,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:06:04,885 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['52957 memcached\n'], []) 2021-12-15 16:06:04,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:06:04,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:06:05,036 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['52957 memcached\n'], []) 2021-12-15 16:06:05,036 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:06:05,039 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:06:05,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:05,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:06:05,542 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:05,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:06:05,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:06:05,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:06:05,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:06:05,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:06:05,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:06:05,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:06:05,687 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:06:05,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:06:05,778 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['102355 memcached\n'], []) 2021-12-15 16:06:05,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:06:05,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:06:05,931 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['102355 memcached\n'], []) 2021-12-15 16:06:05,933 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:06:05,934 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:06:06,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:06,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:06:06,542 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:06,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:06:06,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:06:06,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:06:06,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:06:06,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:06:06,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:06:06,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:06:06,789 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:06:06,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:06:06,841 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['131065 memcached\n'], []) 2021-12-15 16:06:06,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:06:06,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:06:06,983 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['131065 memcached\n'], []) 2021-12-15 16:06:06,983 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:06:06,986 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:06:07,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:07,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:06:07,438 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:07,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:06:07,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:06:07,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:06:07,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:06:07,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:06:07,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:06:07,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:06:07,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:06:07,596 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:06:07,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:06:07,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:07,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:07,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:07,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:07,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:07,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:07,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:08,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:08,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:08,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:08,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:08,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,262 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:08,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:08,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:08,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:08,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:06:08,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:06:08,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:08,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:06:08,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:06:08,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:08,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:06:08,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:06:08,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:06:08,743 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:06:08,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:06:08,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:08,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:06:08,790 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:06:08,790 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:06:08,793 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:06:08,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:09,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:06:09,312 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:09,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:06:09,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:06:09,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:06:09,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:06:09,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:06:09,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:06:09,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:06:09,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:06:09,453 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:06:09,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:06:09,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:09,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:09,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:09,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:09,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:09,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:09,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:09,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:09,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:09,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:09,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:09,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:09,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:10,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:10,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:10,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:10,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:06:10,108 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:06:10,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:06:10,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:10,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:06:10,160 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:06:10,161 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:06:10,163 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:06:10,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:10,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:06:10,631 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:10,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:06:10,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:06:10,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:06:10,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:06:10,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:06:10,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:06:10,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:06:10,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:06:10,767 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:06:10,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:06:10,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:10,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:10,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:10,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:10,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:10,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:10,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:11,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:11,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:06:11,042 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:06:11,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:06:11,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:11,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:06:11,088 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:06:11,088 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:06:11,092 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:06:11,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:11,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:06:11,617 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:11,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:06:11,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:06:11,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:06:11,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:06:11,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:06:11,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:06:11,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:06:11,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:06:11,759 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:06:11,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:06:11,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:11,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:11,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:11,928 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:06:11,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:12,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:12,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:12,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:06:12,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:12,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:12,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:12,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:06:12,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:12,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:06:12,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:12,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:06:12,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:06:12,351 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:06:12,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:06:12,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:06:12,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:06:12,400 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:06:12,400 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:06:12,400 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:06:12,403 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:06:12,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:06:12,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:06:12,895 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:06:12,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:06:12,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:06:12,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:06:12,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:06:12,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:06:12,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:06:13,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:06:13,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:13,288 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 16:06:13,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:13,295 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:15,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:15,395 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:15,401 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:15,401 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:17,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:17,555 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:17,559 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:17,561 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:19,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:19,736 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:19,740 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:19,740 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:21,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:21,914 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:21,917 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:21,918 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:23,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:24,092 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:24,098 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:24,098 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:26,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:26,247 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:26,253 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:26,253 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:28,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:28,427 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:28,431 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:28,431 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:30,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:30,585 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:30,591 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:30,591 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:32,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:32,759 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:06:32,765 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:32,765 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:34,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:34,920 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:34,920 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:36,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:37,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:37,153 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:39,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:39,348 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:39,349 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:41,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:41,526 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:41,526 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:43,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:43,746 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:43,746 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:45,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:45,964 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:45,964 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:47,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:48,154 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:48,154 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:50,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:50,349 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:50,351 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:52,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:52,548 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:52,548 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:54,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:54,733 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:54,733 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:56,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:56,890 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:56,890 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:06:58,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:06:59,095 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:06:59,095 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:01,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:01,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:01,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:03,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:03,529 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:03,529 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:05,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:05,736 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:05,736 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:07,737 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:07,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:07,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:09,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:10,157 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:10,158 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:12,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:12,410 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:12,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:14,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:14,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:14,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:16,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:16,829 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:16,831 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:18,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:19,039 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:19,039 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:21,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:21,242 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:21,242 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:23,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:23,505 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:23,506 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:25,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:25,678 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:25,678 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:27,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:27,917 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:27,917 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:07:29,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:30,253 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:07:30,253 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 77.8539998531. 2021-12-15 16:07:30,253 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:07:30,256 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:07:30,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:07:30,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:07:30,891 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:07:30,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:07:30,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:07:30,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:07:30,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:07:30,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:07:31,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:07:31,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:07:31,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:31,200 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:07:31,201 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.947999954224. 2021-12-15 16:07:31,201 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:07:31,204 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:07:31,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:07:31,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:07:31,703 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:07:31,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:07:31,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:07:31,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:07:31,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:07:31,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:07:31,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:07:31,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:07:31,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:32,059 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:07:32,059 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.858000040054. 2021-12-15 16:07:32,059 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:07:32,062 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:07:32,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:07:32,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:07:32,417 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:07:32,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:07:32,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:07:32,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:07:32,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:07:32,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:07:32,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:07:32,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:07:32,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:32,721 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:32,723 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:07:34,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:34,871 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:34,871 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:07:36,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:37,033 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:37,033 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:07:39,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:39,201 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:39,201 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:07:41,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:41,368 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:07:41,368 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:07:43,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:07:43,528 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:07:43,529 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.4680001736. ############################################################ # # RAM FootPrint: 5628.10546875 # ############################################################ 2021-12-15 16:07:43,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:07:43,565 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 83 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-15 16:09:06,571 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:09:06,571 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:09:11,572 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:09:11,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:12,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:09:12,523 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:12,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:09:12,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:09:12,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:09:12,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:09:12,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:09:12,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:09:12,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:09:12,664 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:09:12,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:09:12,766 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['87477 memcached\n'], []) 2021-12-15 16:09:12,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:09:12,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:09:12,871 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['87477 memcached\n'], []) 2021-12-15 16:09:12,871 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:09:12,874 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:09:13,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:13,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:09:13,513 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:13,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:09:13,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:09:13,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:09:13,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:09:13,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:09:13,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:09:13,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:09:13,720 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:09:13,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:09:13,796 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['53796 memcached\n'], []) 2021-12-15 16:09:13,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:09:13,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:09:13,989 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['53796 memcached\n'], []) 2021-12-15 16:09:13,989 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:09:13,992 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:09:14,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:14,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:09:14,493 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:14,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:09:14,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:09:14,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:09:14,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:09:14,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:09:14,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:09:14,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:09:14,635 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:09:14,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:09:14,690 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['103152 memcached\n'], []) 2021-12-15 16:09:14,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:09:14,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:09:14,828 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['103152 memcached\n'], []) 2021-12-15 16:09:14,828 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:09:14,831 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:09:14,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:15,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:09:15,309 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:15,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:09:15,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:09:15,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:09:15,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:09:15,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:09:15,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:09:15,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:09:15,446 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:09:15,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:09:15,493 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['132145 memcached\n'], []) 2021-12-15 16:09:15,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:09:15,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:09:15,640 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['132145 memcached\n'], []) 2021-12-15 16:09:15,640 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:09:15,642 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:09:15,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:16,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:09:16,104 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:16,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:09:16,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:09:16,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:09:16,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:09:16,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:09:16,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:09:16,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:09:16,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:09:16,262 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:09:16,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:09:16,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:16,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:16,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:16,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:16,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:16,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:16,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:16,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:16,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:16,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:16,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:16,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:16,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:16,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:16,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:16,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:16,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:17,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:17,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:09:17,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:09:17,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:17,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:17,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:09:17,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:09:17,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:17,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:17,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:09:17,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:09:17,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:09:17,388 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:09:17,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:09:17,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:17,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:09:17,434 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:09:17,434 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:09:17,437 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:09:17,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:17,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:09:17,953 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:17,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:09:17,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:09:17,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:09:18,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:09:18,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:09:18,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:09:18,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:09:18,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:09:18,099 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:09:18,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:09:18,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:18,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:18,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:18,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:18,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:18,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:18,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:18,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:18,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:18,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:18,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:18,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:18,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:18,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:18,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:18,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:18,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:09:18,694 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:09:18,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:09:18,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:18,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:09:18,744 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:09:18,744 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:09:18,747 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:09:18,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:19,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:09:19,223 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:19,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:09:19,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:09:19,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:09:19,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:09:19,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:09:19,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:09:19,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:09:19,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:09:19,358 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:09:19,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:09:19,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:19,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:19,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:19,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:19,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:19,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:19,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:19,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:19,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:09:19,688 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:09:19,688 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:09:19,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:19,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:09:19,739 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:09:19,740 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:09:19,742 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:09:19,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:20,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:09:20,267 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:20,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:09:20,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:09:20,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:09:20,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:09:20,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:09:20,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:09:20,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:09:20,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:09:20,381 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:09:20,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:09:20,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:20,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:20,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:20,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:09:20,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:20,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:20,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:20,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:09:20,737 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:20,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:20,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:20,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:09:20,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:20,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:09:20,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:20,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:09:20,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:09:20,901 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:09:20,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:09:20,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:09:20,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:09:20,934 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:09:20,936 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:09:20,936 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:09:20,937 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:09:21,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:09:21,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:09:21,464 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:09:21,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:09:21,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:09:21,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:09:21,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:09:21,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:09:21,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:09:21,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:09:21,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:21,951 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 16:09:21,956 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:21,956 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:23,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:24,062 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:24,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:24,069 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:26,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:26,187 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:26,190 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:26,191 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:28,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:28,358 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:28,362 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:28,362 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:30,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:30,539 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:30,545 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:30,545 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:32,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:32,721 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:32,726 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:32,726 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:34,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:34,858 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:34,862 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:34,862 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:36,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:37,039 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:37,045 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:37,045 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:39,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:39,226 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:39,230 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:39,232 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:41,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:41,398 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:09:41,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:41,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:43,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:43,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:43,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:45,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:45,881 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:45,881 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:47,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:48,105 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:48,105 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:50,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:50,298 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:50,298 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:52,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:52,500 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:52,500 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:54,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:54,674 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:54,676 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:56,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:56,880 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:56,880 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:09:58,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:09:59,112 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:09:59,114 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:01,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:01,338 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:01,338 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:03,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:03,503 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:03,503 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:05,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:05,710 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:05,710 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:07,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:07,938 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:07,938 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:09,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:10,144 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:10,144 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:12,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:12,351 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:12,351 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:14,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:14,585 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:14,585 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:16,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:16,835 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:16,835 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:18,836 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:19,066 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:19,066 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:21,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:21,295 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:21,295 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:23,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:23,540 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:23,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:25,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:25,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:25,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:27,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:28,005 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:28,005 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:30,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:30,183 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:30,183 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:32,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:32,392 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:32,394 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:34,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:34,617 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:34,617 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:10:36,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:36,835 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:10:36,836 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 75.9009997845. 2021-12-15 16:10:36,836 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:10:36,839 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:10:37,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:10:37,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:10:37,596 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:10:37,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:10:37,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:10:37,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:10:37,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:10:37,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:10:37,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:10:37,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:10:38,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:38,207 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:10:38,207 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.37000012398. 2021-12-15 16:10:38,209 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:10:38,210 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:10:38,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:10:38,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:10:38,704 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:10:38,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:10:38,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:10:38,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:10:38,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:10:38,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:10:38,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:10:38,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:10:38,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:39,017 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:10:39,017 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.80899977684. 2021-12-15 16:10:39,019 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:10:39,020 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:10:39,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:10:39,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:10:39,381 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:10:39,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:10:39,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:10:39,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:10:39,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:10:39,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:10:39,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:10:39,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:10:39,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:39,654 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:39,654 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:10:41,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:41,821 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:41,821 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:10:43,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:43,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:43,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:10:45,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:46,151 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:46,151 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:10:48,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:48,345 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:10:48,345 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:10:50,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:10:50,503 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:10:50,503 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.4849998951. ############################################################ # # RAM FootPrint: 5585.27734375 # ############################################################ 2021-12-15 16:10:50,539 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:10:50,539 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 81 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-15 16:12:11,545 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:12:11,545 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:12:16,546 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:12:16,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:16,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:12:17,088 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:17,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:12:17,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:12:17,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:12:17,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:12:17,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:12:17,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:12:17,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:12:17,256 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:12:17,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:12:17,342 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['88726 memcached\n'], []) 2021-12-15 16:12:17,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:12:17,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:12:17,470 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['88726 memcached\n'], []) 2021-12-15 16:12:17,470 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:12:17,473 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:12:17,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:17,857 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:12:17,953 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:17,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:12:17,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:12:17,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:12:18,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:12:18,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:12:18,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:12:18,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:12:18,066 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:12:18,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:12:18,124 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['54629 memcached\n'], []) 2021-12-15 16:12:18,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:12:18,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:12:18,276 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['54629 memcached\n'], []) 2021-12-15 16:12:18,278 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:12:18,280 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:12:18,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:18,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:12:18,782 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:18,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:12:18,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:12:18,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:12:18,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:12:18,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:12:18,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:12:18,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:12:18,907 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:12:18,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:12:18,960 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['103935 memcached\n'], []) 2021-12-15 16:12:18,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:12:19,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:12:19,105 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['103935 memcached\n'], []) 2021-12-15 16:12:19,105 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:12:19,108 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:12:19,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:19,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:12:19,561 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:19,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:12:19,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:12:19,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:12:19,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:12:19,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:12:19,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:12:19,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:12:19,674 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:12:19,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:12:19,733 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['133291 memcached\n'], []) 2021-12-15 16:12:19,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:12:19,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:12:19,835 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['133291 memcached\n'], []) 2021-12-15 16:12:19,835 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:12:19,841 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:12:19,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:20,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:12:20,263 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:20,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:12:20,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:12:20,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:12:20,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:12:20,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:12:20,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:12:20,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:12:20,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:12:20,385 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:12:20,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:12:20,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:20,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:20,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:20,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:20,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:20,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:20,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:20,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:20,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:20,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:20,737 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:20,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:20,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:20,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:20,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:20,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:20,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:20,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:20,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:12:20,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:12:21,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:21,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:21,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:12:21,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:12:21,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:21,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:21,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:12:21,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:12:21,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:12:21,240 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:12:21,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:12:21,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:21,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:12:21,292 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:12:21,292 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:12:21,295 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:12:21,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:21,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:12:21,812 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:21,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:12:21,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:12:21,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:12:21,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:12:21,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:12:21,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:12:21,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:12:21,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:12:21,957 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:12:21,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:12:22,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:22,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:22,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:22,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:22,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:22,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:22,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:22,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:22,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:22,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:22,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:22,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:22,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:22,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:22,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:22,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:22,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:12:22,627 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:12:22,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:12:22,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:22,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:12:22,680 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:12:22,680 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:12:22,681 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:12:22,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:23,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:12:23,181 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:23,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:12:23,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:12:23,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:12:23,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:12:23,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:12:23,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:12:23,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:12:23,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:12:23,335 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:12:23,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:12:23,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:23,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:23,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:23,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:23,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:23,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:23,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:23,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:23,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:12:23,647 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:12:23,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:12:23,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:23,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:12:23,694 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:12:23,694 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:12:23,697 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:12:23,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:24,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:12:24,236 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:24,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:12:24,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:12:24,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:12:24,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:12:24,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:12:24,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:12:24,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:12:24,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:12:24,355 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:12:24,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:12:24,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:24,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:24,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:24,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:12:24,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:24,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:24,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:24,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:12:24,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:24,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:24,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:24,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:12:24,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:24,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:12:24,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:24,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:12:24,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:12:24,966 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:12:24,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:12:24,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:12:24,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:12:25,009 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:12:25,009 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:12:25,009 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:12:25,012 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:12:25,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:12:25,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:12:25,509 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:12:25,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:12:25,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:12:25,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:12:25,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:12:25,573 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:12:25,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:12:25,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:12:25,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:26,796 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 16:12:26,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:26,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:28,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:28,992 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:28,997 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:28,997 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:30,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:31,173 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:31,177 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:31,177 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:33,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:33,355 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:33,359 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:33,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:35,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:35,457 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:35,463 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:35,463 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:37,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:37,624 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:37,630 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:37,630 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:39,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:39,799 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:39,805 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:39,805 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:41,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:41,986 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:41,993 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:41,993 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:43,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:44,163 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:12:44,167 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:44,167 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:46,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:46,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:46,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:48,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:48,559 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:48,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:50,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:50,770 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:50,770 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:52,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:52,967 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:52,967 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:54,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:55,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:55,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:57,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:57,410 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:57,410 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:12:59,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:12:59,588 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:12:59,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:01,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:01,809 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:01,809 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:03,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:04,036 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:04,036 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:06,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:06,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:06,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:08,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:08,388 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:08,388 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:10,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:10,598 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:10,598 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:12,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:12,821 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:12,821 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:14,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:15,042 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:15,043 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:17,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:17,204 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:17,204 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:19,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:19,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:19,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:21,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:21,621 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:21,621 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:23,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:23,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:23,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:25,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:26,128 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:26,128 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:28,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:28,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:28,359 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:30,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:30,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:30,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:32,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:32,826 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:32,826 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:34,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:35,049 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:35,049 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:37,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:37,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:37,306 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:13:39,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:39,526 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:13:39,526 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 74.5169999599. 2021-12-15 16:13:39,526 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:13:39,529 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:13:39,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:13:40,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:13:40,115 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:13:40,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:13:40,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:13:40,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:13:40,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:13:40,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:13:40,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:13:40,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:13:40,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:40,371 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:13:40,371 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.845999956131. 2021-12-15 16:13:40,372 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:13:40,375 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:13:40,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:13:40,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:13:40,862 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:13:40,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:13:40,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:13:40,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:13:40,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:13:40,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:13:40,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:13:40,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:13:41,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:41,217 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:13:41,217 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.84500002861. 2021-12-15 16:13:41,220 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:13:41,221 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:13:41,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:13:41,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:13:41,559 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:13:41,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:13:41,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:13:41,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:13:41,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:13:41,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:13:41,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:13:41,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:13:41,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:41,845 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:41,845 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:13:43,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:44,015 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:44,015 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:13:46,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:46,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:13:46,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:13:48,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:13:48,483 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:13:48,484 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.26399993896. ############################################################ # # RAM FootPrint: 5553.22265625 # ############################################################ 2021-12-15 16:13:48,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:13:48,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-15 16:14:55,519 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:14:55,520 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:15:00,523 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:15:01,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:01,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:15:01,576 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:01,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:15:01,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:15:01,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:15:01,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:15:01,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:15:01,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:15:01,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:15:01,813 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:15:01,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:15:01,871 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['89962 memcached\n'], []) 2021-12-15 16:15:01,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:15:01,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:15:02,065 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['89962 memcached\n'], []) 2021-12-15 16:15:02,065 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:15:02,069 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:15:02,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:02,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:15:02,630 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:02,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:15:02,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:15:02,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:15:02,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:15:02,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:15:02,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:15:02,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:15:02,845 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:15:02,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:15:02,933 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['55452 memcached\n'], []) 2021-12-15 16:15:02,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:15:03,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:15:03,144 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['55452 memcached\n'], []) 2021-12-15 16:15:03,144 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:15:03,148 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:15:03,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:03,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:15:03,661 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:03,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:15:03,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:15:03,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:15:03,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:15:03,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:15:03,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:15:03,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:15:03,812 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:15:03,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:15:03,891 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['104714 memcached\n'], []) 2021-12-15 16:15:03,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:15:03,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:15:04,026 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['104714 memcached\n'], []) 2021-12-15 16:15:04,026 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:15:04,030 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:15:04,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:04,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:15:04,493 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:04,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:15:04,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:15:04,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:15:04,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:15:04,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:15:04,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:15:04,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:15:04,615 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:15:04,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:15:04,668 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['134445 memcached\n'], []) 2021-12-15 16:15:04,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:15:04,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:15:04,819 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['134445 memcached\n'], []) 2021-12-15 16:15:04,819 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:15:04,822 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:15:04,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:05,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:15:05,243 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:05,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:15:05,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:15:05,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:15:05,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:15:05,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:15:05,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:15:05,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:15:05,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:15:05,427 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:15:05,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:15:05,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:05,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:05,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:05,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:05,655 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:05,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:05,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:05,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:05,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:05,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:05,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:05,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:05,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:06,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:06,086 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:06,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:06,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:06,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:06,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:15:06,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:15:06,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:06,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:06,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:15:06,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:15:06,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:06,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:06,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:15:06,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:15:06,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:15:06,608 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:15:06,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:15:06,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:06,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:15:06,661 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:15:06,661 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:15:06,664 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:15:06,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:07,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:15:07,168 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:07,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:15:07,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:15:07,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:15:07,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:15:07,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:15:07,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:15:07,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:15:07,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:15:07,302 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:15:07,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:15:07,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:07,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:07,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:07,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:07,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:07,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:07,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:07,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:07,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:07,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:07,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:07,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:07,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:07,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:07,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:07,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:07,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:15:07,921 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:15:07,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:15:07,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:07,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:15:07,961 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:15:07,961 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:15:07,964 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:15:08,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:08,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:15:08,447 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:08,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:15:08,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:15:08,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:15:08,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:15:08,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:15:08,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:15:08,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:15:08,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:15:08,594 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:15:08,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:15:08,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:08,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:08,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:08,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:08,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:08,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:08,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:08,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:08,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:15:08,948 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:15:08,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:15:08,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:08,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:15:09,009 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:15:09,009 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:15:09,012 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:15:09,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:09,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:15:09,493 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:09,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:15:09,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:15:09,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:15:09,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:15:09,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:15:09,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:15:09,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:15:09,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:15:09,608 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:15:09,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:15:09,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:09,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:09,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:09,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:15:09,801 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:09,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:09,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:09,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:15:09,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:09,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:10,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:10,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:15:10,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:10,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:15:10,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:10,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:15:10,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:15:10,151 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:15:10,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:15:10,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:15:10,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:15:10,197 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:15:10,197 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:15:10,197 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:15:10,200 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:15:10,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:15:11,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:15:11,901 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:15:11,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:15:11,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:15:11,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:15:11,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:15:11,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:15:11,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:15:12,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:15:12,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:12,194 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:12,200 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:12,200 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:14,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:14,336 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:14,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:14,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:16,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:16,512 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:16,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:16,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:18,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:18,615 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:18,621 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:18,621 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:20,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:20,795 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:20,799 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:20,799 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:22,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:22,960 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:22,966 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:22,967 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:24,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:25,122 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:25,128 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:25,128 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:27,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:27,285 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:27,290 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:27,290 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:29,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:29,457 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:29,461 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:29,461 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:31,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:31,568 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:15:31,572 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:31,572 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:33,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:33,723 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:33,723 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:35,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:35,953 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:35,953 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:37,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:38,176 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:38,176 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:40,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:40,400 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:40,400 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:42,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:42,611 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:42,611 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:44,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:44,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:44,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:46,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:47,040 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:47,040 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:49,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:49,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:49,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:51,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:51,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:51,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:53,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:53,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:53,565 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:55,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:55,766 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:55,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:57,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:15:57,924 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:15:57,924 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:15:59,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:00,069 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:00,069 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:02,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:02,301 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:02,302 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:04,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:04,558 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:04,559 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:06,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:06,729 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:06,729 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:08,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:08,940 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:08,940 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:10,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:11,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:11,188 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:13,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:13,539 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:13,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:15,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:15,757 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:15,757 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:17,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:18,062 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:18,062 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:20,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:20,339 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:20,339 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:22,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:22,572 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:22,572 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:16:24,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:25,108 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:16:25,108 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 74.9110000134. 2021-12-15 16:16:25,109 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:16:25,111 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:16:25,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:16:25,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:16:25,595 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:16:25,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:16:25,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:16:25,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:16:25,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:16:25,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:16:25,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:16:25,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:16:25,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:25,885 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:16:25,891 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.78200006485. 2021-12-15 16:16:25,891 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:16:25,894 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:16:26,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:16:26,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:16:26,414 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:16:26,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:16:26,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:16:26,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:16:26,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:16:26,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:16:26,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:16:26,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:16:26,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:26,683 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:16:26,683 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.792000055313. 2021-12-15 16:16:26,683 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:16:26,686 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:16:26,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:16:26,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:16:27,026 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:16:27,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:16:27,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:16:27,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:16:27,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:16:27,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:16:27,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:16:27,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:16:27,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:27,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:27,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:16:29,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:29,464 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:29,466 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:16:31,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:31,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:16:31,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:16:33,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:16:33,772 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:16:33,772 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.08899998665. ############################################################ # # RAM FootPrint: 5545.50390625 # ############################################################ 2021-12-15 16:16:33,805 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:16:33,806 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 60 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-15 16:17:33,812 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:17:33,812 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:17:38,815 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:17:39,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:40,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:17:40,144 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:40,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:17:40,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:17:40,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:17:40,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:17:40,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:17:40,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:17:40,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:17:40,348 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:17:40,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:17:40,405 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['91111 memcached\n'], []) 2021-12-15 16:17:40,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:17:40,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:17:40,657 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['91111 memcached\n'], []) 2021-12-15 16:17:40,657 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:17:40,660 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:17:41,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:43,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:17:43,177 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:43,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:17:43,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:17:43,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:17:43,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:17:43,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:17:43,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:17:43,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:17:43,332 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:17:43,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:17:43,421 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['56258 memcached\n'], []) 2021-12-15 16:17:43,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:17:43,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:17:43,572 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['56258 memcached\n'], []) 2021-12-15 16:17:43,573 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:17:43,576 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:17:43,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:43,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:17:44,082 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:44,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:17:44,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:17:44,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:17:44,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:17:44,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:17:44,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:17:44,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:17:44,234 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:17:44,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:17:44,313 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['105477 memcached\n'], []) 2021-12-15 16:17:44,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:17:44,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:17:44,438 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['105477 memcached\n'], []) 2021-12-15 16:17:44,438 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:17:44,441 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:17:45,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:45,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:17:45,762 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:45,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:17:45,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:17:45,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:17:45,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:17:45,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:17:45,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:17:45,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:17:45,920 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:17:45,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:17:45,976 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['135531 memcached\n'], []) 2021-12-15 16:17:45,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:17:46,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:17:46,128 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['135531 memcached\n'], []) 2021-12-15 16:17:46,128 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:17:46,131 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:17:46,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:46,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:17:46,578 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:46,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:17:46,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:17:46,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:17:46,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:17:46,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:17:46,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:17:46,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:17:46,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:17:46,726 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:17:46,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:17:46,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:46,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:46,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:46,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:46,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:47,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:47,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:47,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:47,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:47,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:47,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:17:47,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:17:47,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,706 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:17:47,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:17:47,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:47,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:47,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:17:47,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:17:47,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:17:47,963 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:17:47,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:17:47,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:48,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:17:48,025 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:17:48,025 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:17:48,028 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:17:48,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:48,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:17:48,503 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:48,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:17:48,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:17:48,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:17:48,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:17:48,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:17:48,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:17:48,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:17:48,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:17:48,614 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:17:48,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:17:48,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:48,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:48,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:48,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:48,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:48,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:48,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:48,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:48,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:48,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:48,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:49,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:49,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:49,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:49,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:49,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:49,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:49,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:49,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:17:49,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:17:49,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:17:49,154 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:17:49,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:17:49,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:49,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:17:49,190 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:17:49,190 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:17:49,194 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:17:49,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:49,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:17:49,694 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:49,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:17:49,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:17:49,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:17:49,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:17:49,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:17:49,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:17:49,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:17:49,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:17:49,841 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:17:49,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:17:49,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:49,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:49,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:49,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:50,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:50,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:50,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:50,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:50,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:50,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:50,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:50,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:50,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:17:50,161 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:17:50,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:17:50,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:50,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:17:50,203 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:17:50,203 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:17:50,206 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:17:50,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:50,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:17:50,724 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:50,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:17:50,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:17:50,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:17:50,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:17:50,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:17:50,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:17:50,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:17:50,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:17:50,862 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:17:50,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:17:50,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:50,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:51,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:51,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:17:51,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:51,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:51,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:51,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:17:51,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:51,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:51,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:51,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:17:51,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:51,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:17:51,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:51,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:17:51,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:17:51,536 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:17:51,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:17:51,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:17:51,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:17:51,589 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:17:51,589 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:17:51,589 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:17:51,592 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:17:51,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:17:51,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:17:51,983 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:17:51,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:17:52,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:17:52,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:17:52,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:17:52,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:17:52,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:17:52,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:17:52,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:17:52,273 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:17:52,278 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:17:52,278 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:17:54,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:17:54,446 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:17:54,450 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:17:54,450 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:17:56,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:17:56,618 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:17:56,622 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:17:56,622 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:17:58,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:17:58,789 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:17:58,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:17:58,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:00,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:00,976 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:18:00,982 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:00,982 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:02,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:03,180 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:18:03,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:03,186 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:05,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:05,358 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:18:05,364 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:05,364 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:07,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:07,539 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:07,539 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:09,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:09,767 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:09,769 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:11,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:11,969 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:11,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:13,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:14,200 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:14,200 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:16,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:16,430 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:16,430 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:18,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:18,667 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:18,667 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:20,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:20,867 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:20,867 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:22,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:23,078 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:23,079 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:25,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:25,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:25,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:27,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:27,382 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:27,382 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:29,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:29,555 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:29,555 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:31,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:31,720 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:31,720 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:33,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:33,880 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:33,880 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:35,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:36,039 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:36,039 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:38,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:38,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:38,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:40,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:40,348 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:40,349 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:42,349 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:42,513 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:42,513 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:44,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:44,677 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:44,677 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:46,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:46,854 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:46,855 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:48,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:49,049 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:49,051 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:51,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:51,236 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:51,236 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:53,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:53,417 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:53,417 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:55,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:55,619 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:55,621 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:57,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:57,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:57,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:18:59,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:18:59,966 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:18:59,966 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:19:01,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:02,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:19:02,194 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:19:04,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:04,400 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:19:04,401 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:19:06,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:06,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:19:06,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:19:08,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:08,829 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:19:08,829 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:19:10,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:10,996 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:19:10,996 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 79.4070000648. 2021-12-15 16:19:10,999 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:19:11,000 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:19:11,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:19:11,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:19:11,657 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:19:11,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:19:11,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:19:11,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:19:11,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:19:11,734 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:19:11,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:19:11,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:19:11,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:11,992 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:19:11,993 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.994000196457. 2021-12-15 16:19:11,993 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:19:11,994 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:19:12,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:19:12,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:19:12,486 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:19:12,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:19:12,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:19:12,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:19:12,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:19:12,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:19:12,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:19:12,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:19:12,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:12,802 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:19:12,802 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.80999994278. 2021-12-15 16:19:12,802 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:19:12,806 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:19:12,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:19:13,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:19:13,259 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:19:13,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:19:13,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:19:13,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:19:13,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:19:13,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:19:13,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:19:13,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:19:13,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:19:13,618 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:19:13,618 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.815999984741. ############################################################ # # RAM FootPrint: 5528.5 # ############################################################ 2021-12-15 16:19:13,657 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:19:13,657 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 98 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-15 16:20:51,664 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:20:51,665 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:20:56,667 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:20:56,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:20:57,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:20:57,338 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:20:57,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:20:57,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:20:57,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:20:57,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:20:57,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:20:57,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:20:57,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:20:57,559 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:20:57,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:20:57,680 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['92339 memcached\n'], []) 2021-12-15 16:20:57,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:20:57,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:20:57,880 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['92339 memcached\n'], []) 2021-12-15 16:20:57,881 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:20:57,884 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:20:58,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:20:58,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:20:58,543 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:20:58,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:20:58,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:20:58,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:20:58,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:20:58,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:20:58,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:20:58,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:20:58,802 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:20:58,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:20:58,892 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['57070 memcached\n'], []) 2021-12-15 16:20:58,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:20:59,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:20:59,108 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['57070 memcached\n'], []) 2021-12-15 16:20:59,108 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:20:59,111 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:20:59,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:20:59,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:20:59,624 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:20:59,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:20:59,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:20:59,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:20:59,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:20:59,709 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:20:59,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:20:59,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:20:59,766 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:20:59,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:20:59,844 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['106269 memcached\n'], []) 2021-12-15 16:20:59,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:20:59,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:20:59,977 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['106269 memcached\n'], []) 2021-12-15 16:20:59,977 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:20:59,980 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:21:00,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:00,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:21:00,579 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:00,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:21:00,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:21:00,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:21:00,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:21:00,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:21:00,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:21:00,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:21:00,714 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:21:00,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:21:00,765 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['136611 memcached\n'], []) 2021-12-15 16:21:00,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:21:00,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:21:00,907 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['136611 memcached\n'], []) 2021-12-15 16:21:00,907 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:21:00,911 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:21:01,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:01,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:21:01,364 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:01,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:21:01,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:21:01,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:21:01,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:21:01,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:21:01,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:21:01,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:21:01,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:21:01,513 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:21:01,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:21:01,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:01,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:01,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:01,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:01,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:01,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:01,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:01,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:01,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:01,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:02,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:02,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:02,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:02,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:02,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:02,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:21:02,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:21:02,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:21:02,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:21:02,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:02,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:21:02,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:21:02,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:21:02,859 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:21:02,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:21:02,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:02,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:21:02,907 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:21:02,907 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:21:02,911 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:21:03,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:04,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:21:04,190 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:04,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:21:04,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:21:04,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:21:04,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:21:04,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:21:04,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:21:04,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:21:04,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:21:04,334 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:21:04,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:21:04,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:04,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:04,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:04,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:04,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:04,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:04,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:04,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:04,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:04,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:04,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:04,854 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:04,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:04,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:04,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:04,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:05,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:05,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:05,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:05,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:05,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:21:05,088 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:21:05,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:21:05,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:05,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:21:05,125 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:21:05,125 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:21:05,128 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:21:05,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:05,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:21:05,612 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:05,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:21:05,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:21:05,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:21:05,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:21:05,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:21:05,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:21:05,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:21:05,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:21:05,769 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:21:05,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:21:05,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:05,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:05,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:05,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:05,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:05,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:06,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:06,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:06,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:06,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:06,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:06,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:06,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:21:06,134 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:21:06,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:21:06,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:06,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:21:06,171 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:21:06,171 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:21:06,174 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:21:06,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:06,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:21:06,667 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:06,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:21:06,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:21:06,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:21:06,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:21:06,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:21:06,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:21:06,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:21:06,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:21:06,782 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:21:06,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:21:06,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:06,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:06,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:06,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:21:06,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:07,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:07,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:07,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:21:07,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:07,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:07,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:07,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:21:07,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:07,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:07,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:07,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:21:07,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:07,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:21:07,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:07,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:21:07,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:21:07,473 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:21:07,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:21:07,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:21:07,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:21:07,516 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:21:07,516 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:21:07,517 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:21:07,519 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:21:07,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:21:07,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:21:08,065 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:21:08,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:21:08,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:21:08,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:21:08,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:21:08,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:21:08,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:21:08,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:21:08,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:08,358 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:08,364 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:08,364 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:10,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:10,542 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:10,548 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:10,548 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:12,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:12,724 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:12,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:12,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:14,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:14,901 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:14,907 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:14,907 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:16,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:17,082 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:17,089 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:17,091 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:19,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:19,282 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:19,286 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:19,286 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:21,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:21,460 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:21,464 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:21,466 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:23,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:23,628 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:23,634 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:23,634 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:25,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:25,815 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:25,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:25,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:27,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:28,003 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:21:28,009 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:28,009 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:30,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:30,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:30,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:32,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:32,385 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:32,387 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:34,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:34,552 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:34,552 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:36,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:36,795 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:36,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:38,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:38,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:38,979 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:40,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:41,209 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:41,209 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:43,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:43,375 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:43,375 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:45,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:45,569 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:45,569 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:47,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:47,727 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:47,727 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:49,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:49,878 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:49,878 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:51,880 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:52,030 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:52,030 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:54,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:54,229 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:54,229 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:56,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:56,441 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:56,441 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:21:58,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:21:58,595 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:21:58,595 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:00,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:00,775 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:00,775 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:02,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:02,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:02,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:04,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:05,122 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:05,122 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:07,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:07,283 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:07,283 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:09,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:09,464 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:09,464 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:11,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:11,640 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:11,640 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:13,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:13,823 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:13,825 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:15,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:16,025 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:16,025 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:18,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:18,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:18,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:20,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:20,372 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:20,372 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:22,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:22,561 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:22,561 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:24,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:24,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:24,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:26,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:26,894 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:22:26,895 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:22:28,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:29,144 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:22:29,144 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 81.6260001659. 2021-12-15 16:22:29,144 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:22:29,147 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:22:29,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:22:29,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:22:29,582 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:22:29,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:22:29,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:22:29,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:22:29,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:22:29,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:22:29,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:22:29,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:22:29,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:29,892 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:22:29,894 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.749000072479. 2021-12-15 16:22:29,894 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:22:29,895 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:22:30,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:22:30,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:22:30,384 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:22:30,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:22:30,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:22:30,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:22:30,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:22:30,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:22:30,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:22:30,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:22:30,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:30,703 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:22:30,703 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.80999994278. 2021-12-15 16:22:30,704 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:22:30,707 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:22:30,844 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:22:31,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:22:31,177 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:22:31,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:22:31,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:22:31,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:22:31,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:22:31,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:22:31,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:22:31,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:22:31,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:22:31,436 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:22:31,436 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.732000112534. ############################################################ # # RAM FootPrint: 5506.4765625 # ############################################################ 2021-12-15 16:22:31,469 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:22:31,469 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 77 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-15 16:23:48,473 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:23:48,473 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:23:53,476 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:23:53,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:23:53,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:23:54,035 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:23:54,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:23:54,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:23:54,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:23:54,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:23:54,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:23:54,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:23:54,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:23:54,180 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:23:54,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:23:54,240 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['93652 memcached\n'], []) 2021-12-15 16:23:54,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:23:54,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:23:54,391 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['93652 memcached\n'], []) 2021-12-15 16:23:54,392 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:23:54,395 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:23:54,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:23:56,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:23:56,321 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:23:56,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:23:56,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:23:56,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:23:56,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:23:56,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:23:56,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:23:56,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:23:56,538 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:23:56,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:23:56,607 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['57944 memcached\n'], []) 2021-12-15 16:23:56,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:23:56,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:23:56,757 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['57944 memcached\n'], []) 2021-12-15 16:23:56,757 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:23:56,763 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:23:56,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:23:57,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:23:57,312 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:23:57,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:23:57,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:23:57,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:23:57,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:23:57,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:23:57,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:23:57,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:23:57,464 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:23:57,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:23:57,535 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['107090 memcached\n'], []) 2021-12-15 16:23:57,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:23:57,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:23:57,680 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['107090 memcached\n'], []) 2021-12-15 16:23:57,681 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:23:57,684 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:23:57,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:23:58,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:23:58,180 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:23:58,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:23:58,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:23:58,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:23:58,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:23:58,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:23:58,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:23:58,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:23:58,321 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:23:58,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:23:58,374 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['137720 memcached\n'], []) 2021-12-15 16:23:58,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:23:58,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:23:58,516 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['137720 memcached\n'], []) 2021-12-15 16:23:58,516 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:23:58,519 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:23:58,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:23:58,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:23:58,933 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:23:58,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:23:58,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:23:58,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:23:58,992 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:23:59,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:23:59,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:23:59,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:23:59,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:23:59,088 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:23:59,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:23:59,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:23:59,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:23:59,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:23:59,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:23:59,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:23:59,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:23:59,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:23:59,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:23:59,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:23:59,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:23:59,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:23:59,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:23:59,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:23:59,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:23:59,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:00,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:00,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:24:00,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:24:00,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:00,122 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:00,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:24:00,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:24:00,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:24:00,190 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:24:00,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:24:00,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:00,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:24:00,230 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:24:00,230 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:24:00,233 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:24:00,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:24:00,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:24:00,736 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:24:00,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:24:00,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:24:00,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:24:00,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:24:00,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:24:00,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:24:00,836 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:24:00,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:24:00,868 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:24:00,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:24:00,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:00,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:01,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:01,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:01,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:01,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:01,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:01,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:01,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:01,293 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:01,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:01,364 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:01,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:01,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:01,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:24:01,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:24:01,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:01,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:01,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:24:01,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:24:01,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:24:01,631 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:24:01,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:24:01,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:01,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:24:01,678 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:24:01,678 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:24:01,681 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:24:01,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:24:02,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:24:02,170 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:24:02,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:24:02,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:24:02,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:24:02,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:24:02,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:24:02,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:24:02,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:24:02,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:24:02,282 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:24:02,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:24:02,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:02,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:02,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:02,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:02,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:02,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:02,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:02,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:02,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:02,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:02,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:02,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:02,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:24:02,665 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:24:02,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:24:02,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:02,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:24:02,704 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:24:02,704 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:24:02,707 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:24:02,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:24:03,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:24:03,207 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:24:03,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:24:03,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:24:03,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:24:03,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:24:03,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:24:03,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:24:03,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:24:03,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:24:03,338 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:24:03,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:24:03,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:03,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:03,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:03,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:24:03,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:03,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:03,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:03,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:24:03,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:03,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:03,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:03,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:24:03,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:03,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:03,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:24:03,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:24:03,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:03,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:24:03,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:24:03,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:24:04,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:24:04,030 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:24:04,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:24:04,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:24:04,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:24:04,072 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:24:04,072 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:24:04,072 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:24:04,075 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:24:04,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:24:04,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:24:04,584 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:24:04,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:24:04,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:24:04,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:24:04,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:24:04,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:24:04,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:24:04,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:24:04,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:04,792 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:04,798 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:04,798 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:06,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:06,970 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:06,976 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:06,976 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:08,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:09,154 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:09,158 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:09,158 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:11,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:11,341 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:11,345 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:11,345 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:13,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:13,509 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:13,513 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:13,515 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:15,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:15,684 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:15,690 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:15,690 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:17,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:17,868 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:17,874 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:17,874 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:19,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:20,053 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:24:20,059 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:20,059 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:22,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:22,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:22,242 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:24,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:24,447 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:24,447 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:26,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:26,638 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:26,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:28,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:28,805 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:28,805 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:30,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:30,970 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:30,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:32,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:33,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:33,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:35,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:35,398 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:35,398 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:37,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:37,614 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:37,614 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:39,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:39,815 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:39,815 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:41,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:42,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:42,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:44,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:44,194 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:44,196 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:46,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:46,354 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:46,354 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:48,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:48,516 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:48,516 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:50,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:50,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:50,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:52,673 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:53,147 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:53,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:55,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:55,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:55,358 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:57,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:57,549 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:57,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:24:59,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:24:59,724 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:24:59,724 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:01,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:01,926 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:01,926 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:03,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:04,104 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:04,105 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:06,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:06,269 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:06,269 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:08,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:08,434 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:08,434 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:10,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:10,615 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:10,615 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:12,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:12,776 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:12,776 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:14,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:14,953 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:14,953 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:16,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:17,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:17,118 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:25:19,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:19,269 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:25:19,270 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 75.1970000267. 2021-12-15 16:25:19,272 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:25:19,273 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:25:19,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:25:19,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:25:19,680 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:25:19,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:25:19,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:25:19,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:25:19,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:25:19,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:25:19,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:25:19,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:25:19,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:19,954 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:19,954 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:25:21,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:22,132 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:22,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:25:24,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:24,384 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:24,384 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:25:26,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:26,661 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:25:26,661 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.38999986649. 2021-12-15 16:25:26,664 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:25:26,664 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:25:26,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:25:27,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:25:27,174 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:25:27,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:25:27,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:25:27,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:25:27,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:25:27,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:25:27,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:25:27,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:25:27,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:27,440 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:25:27,440 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.776000022888. 2021-12-15 16:25:27,440 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:25:27,444 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:25:27,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:25:27,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:25:27,785 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:25:27,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:25:27,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:25:27,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:25:27,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:25:27,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:25:27,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:25:27,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:25:27,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:28,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:28,065 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:25:30,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:30,236 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:30,237 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:25:32,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:32,401 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:25:32,401 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:25:34,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:25:34,555 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:25:34,555 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.11599993706. ############################################################ # # RAM FootPrint: 5492.65625 # ############################################################ 2021-12-15 16:25:34,588 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:25:34,588 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-15 16:27:08,592 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:27:08,592 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:27:13,594 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:27:13,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:14,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:27:14,187 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:14,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:27:14,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:27:14,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:27:14,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:27:14,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:27:14,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:27:14,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:27:14,305 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:27:14,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:27:14,358 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['94945 memcached\n'], []) 2021-12-15 16:27:14,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:27:14,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:27:14,493 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['94945 memcached\n'], []) 2021-12-15 16:27:14,493 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:27:14,497 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:27:14,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:15,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:27:15,194 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:15,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:27:15,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:27:15,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:27:15,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:27:15,293 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:27:15,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:27:15,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:27:15,345 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:27:15,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:27:15,405 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['58788 memcached\n'], []) 2021-12-15 16:27:15,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:27:15,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:27:15,677 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['58788 memcached\n'], []) 2021-12-15 16:27:15,677 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:27:15,680 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:27:15,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:16,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:27:16,196 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:16,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:27:16,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:27:16,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:27:16,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:27:16,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:27:16,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:27:16,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:27:16,359 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:27:16,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:27:16,447 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['107894 memcached\n'], []) 2021-12-15 16:27:16,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:27:16,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:27:16,578 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['107894 memcached\n'], []) 2021-12-15 16:27:16,578 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:27:16,581 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:27:16,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:16,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:27:17,065 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:17,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:27:17,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:27:17,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:27:17,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:27:17,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:27:17,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:27:17,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:27:17,250 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:27:17,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:27:17,336 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['138817 memcached\n'], []) 2021-12-15 16:27:17,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:27:17,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:27:17,492 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['138817 memcached\n'], []) 2021-12-15 16:27:17,493 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:27:17,494 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:27:17,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:17,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:27:17,941 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:17,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:27:17,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:27:17,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:27:18,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:27:18,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:27:18,040 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:27:18,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:27:18,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:27:18,095 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:27:18,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:27:18,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:18,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:18,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:18,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:18,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:18,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:18,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:18,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:18,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:18,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:18,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:18,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:18,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:27:18,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:27:19,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:19,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:19,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:27:19,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:27:19,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:19,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:19,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:27:19,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:27:19,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:27:19,221 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:27:19,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:27:19,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:19,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:27:19,266 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:27:19,266 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:27:19,269 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:27:19,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:20,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:27:20,210 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:20,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:27:20,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:27:20,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:27:20,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:27:20,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:27:20,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:27:20,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:27:20,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:27:20,359 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:27:20,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:27:20,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:20,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:20,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:20,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:20,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:20,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:20,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:20,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:20,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:20,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:20,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:20,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:20,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:20,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:21,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:21,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:21,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:21,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:21,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:21,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:21,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:27:21,211 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:27:21,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:27:21,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:21,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:27:21,262 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:27:21,262 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:27:21,263 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:27:21,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:21,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:27:21,736 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:21,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:27:21,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:27:21,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:27:21,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:27:21,811 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:27:21,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:27:21,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:27:21,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:27:21,872 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:27:21,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:27:21,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:21,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:22,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:22,033 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:22,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:22,091 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:22,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:22,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:22,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:22,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:22,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:22,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:22,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:27:22,272 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:27:22,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:27:22,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:22,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:27:22,321 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:27:22,321 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:27:22,323 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:27:22,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:22,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:27:22,874 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:22,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:27:22,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:27:22,910 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:27:22,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:27:22,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:27:22,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:27:22,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:27:22,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:27:23,003 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:27:23,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:27:23,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:23,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:23,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:27:23,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:23,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:23,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:27:23,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:23,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:23,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:27:23,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:23,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:23,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:27:23,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:27:23,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:23,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:27:23,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:27:23,464 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:27:23,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:27:23,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:27:23,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:27:23,496 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:27:23,497 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:27:23,497 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:27:23,500 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:27:23,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:27:24,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:27:24,112 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:27:24,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:27:24,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:27:24,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:27:24,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:27:24,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:27:24,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:27:24,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:27:24,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:24,410 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:24,418 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:24,418 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:26,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:26,591 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:26,596 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:26,596 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:28,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:28,753 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:28,760 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:28,760 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:30,762 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:30,943 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:30,947 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:30,947 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:32,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:33,125 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:33,130 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:33,130 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:35,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:35,236 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:35,240 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:35,240 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:37,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:37,407 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:37,413 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:37,413 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:39,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:39,598 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:39,602 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:39,602 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:41,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:41,726 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:41,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:41,732 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:43,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:43,857 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:43,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:43,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:45,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:46,039 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:46,046 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:46,046 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:48,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:48,226 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:27:48,230 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:48,230 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:50,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:50,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:50,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:52,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:52,694 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:52,696 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:54,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:54,930 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:54,930 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:56,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:57,125 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:57,125 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:27:59,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:27:59,323 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:27:59,325 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:01,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:01,466 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:01,466 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:03,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:03,624 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:03,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:05,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:05,811 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:05,811 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:07,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:08,017 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:08,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:10,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:10,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:10,203 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:12,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:12,384 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:12,384 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:14,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:14,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:14,563 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:16,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:16,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:16,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:18,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:18,970 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:18,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:20,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:21,229 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:21,229 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:23,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:23,444 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:23,444 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:25,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:25,612 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:25,614 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:27,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:27,770 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:27,772 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:29,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:29,934 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:29,934 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:31,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:32,104 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:32,104 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:34,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:34,262 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:34,262 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:36,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:36,437 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:36,437 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:38,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:38,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:38,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:40,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:40,832 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:40,832 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:42,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:43,002 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:28:43,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:28:45,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:45,211 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:28:45,211 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 81.7139999866. 2021-12-15 16:28:45,213 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:28:45,216 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:28:45,553 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:28:47,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:28:47,520 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:28:47,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:28:47,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:28:47,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:28:47,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:28:47,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:28:47,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:28:47,719 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:28:47,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:47,951 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:28:47,953 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.73900008202. 2021-12-15 16:28:47,953 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:28:47,957 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:28:48,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:28:48,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:28:48,431 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:28:48,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:28:48,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:28:48,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:28:48,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:28:48,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:28:48,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:28:48,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:28:48,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:48,727 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:28:48,727 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.774999856949. 2021-12-15 16:28:48,729 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:28:48,730 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:28:48,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:28:49,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:28:49,196 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:28:49,196 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:28:49,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:28:49,234 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:28:49,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:28:49,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:28:49,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:28:49,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:28:49,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:28:49,470 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:28:49,470 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.741999864578. ############################################################ # # RAM FootPrint: 5466.3203125 # ############################################################ 2021-12-15 16:28:49,503 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:28:49,503 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 117 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-15 16:30:46,507 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:30:46,507 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:30:51,509 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:30:51,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:30:52,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:30:52,207 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:30:52,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:30:52,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:30:52,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:30:52,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:30:52,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:30:52,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:30:52,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:30:52,400 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:30:52,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:30:52,460 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['96241 memcached\n'], []) 2021-12-15 16:30:52,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:30:52,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:30:52,683 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['96241 memcached\n'], []) 2021-12-15 16:30:52,683 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:30:52,687 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:30:52,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:30:53,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:30:53,240 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:30:53,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:30:53,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:30:53,303 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:30:53,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:30:53,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:30:53,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:30:53,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:30:53,474 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:30:53,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:30:53,543 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['59715 memcached\n'], []) 2021-12-15 16:30:53,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:30:53,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:30:53,776 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['59715 memcached\n'], []) 2021-12-15 16:30:53,776 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:30:53,780 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:30:53,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:30:54,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:30:54,279 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:30:54,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:30:54,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:30:54,318 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:30:54,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:30:54,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:30:54,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:30:54,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:30:54,398 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:30:54,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:30:54,447 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['108720 memcached\n'], []) 2021-12-15 16:30:54,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:30:54,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:30:54,558 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['108720 memcached\n'], []) 2021-12-15 16:30:54,558 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:30:54,561 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:30:54,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:30:54,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:30:55,029 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:30:55,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:30:55,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:30:55,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:30:55,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:30:55,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:30:55,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:30:55,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:30:55,210 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:30:55,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:30:55,305 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['139982 memcached\n'], []) 2021-12-15 16:30:55,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:30:55,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:30:55,450 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['139982 memcached\n'], []) 2021-12-15 16:30:55,450 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:30:55,453 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:30:55,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:30:55,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:30:55,901 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:30:55,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:30:55,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:30:55,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:30:55,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:30:55,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:30:56,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:30:56,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:30:56,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:30:56,053 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:30:56,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:30:56,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:56,220 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:30:56,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:30:56,293 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,357 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:56,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:30:56,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:30:56,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:56,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:30:56,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:30:56,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:56,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:30:56,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:30:56,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:56,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:30:56,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:30:56,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:56,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:57,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:30:57,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:30:57,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:57,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:57,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:30:57,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:30:57,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:57,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:30:57,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:30:57,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:30:57,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:30:57,391 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:30:57,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:30:57,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:30:57,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:30:57,446 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:30:57,447 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:30:57,450 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:30:57,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:31:00,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:31:00,121 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:31:00,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:31:00,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:31:00,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:31:00,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:31:00,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:31:00,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:31:00,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:31:00,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:31:00,266 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:31:00,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:31:00,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:00,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:00,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:00,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:00,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:00,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:00,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:00,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:00,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:00,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:00,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:00,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:00,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:00,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:00,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:31:00,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:31:00,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:01,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:01,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:31:01,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:31:01,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:31:01,117 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:31:01,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:31:01,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:01,153 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:31:01,168 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:31:01,168 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:31:01,171 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:31:01,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:31:01,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:31:01,661 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:31:01,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:31:01,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:31:01,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:31:01,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:31:01,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:31:01,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:31:01,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:31:01,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:31:01,802 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:31:01,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:31:01,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:01,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:01,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:01,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:01,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:02,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:02,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:02,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:02,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:02,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:02,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:02,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:02,239 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:31:02,256 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:31:02,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:31:02,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:02,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:31:02,305 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:31:02,305 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:31:02,309 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:31:02,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:31:02,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:31:02,834 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:31:02,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:31:02,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:31:02,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:31:02,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:31:02,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:31:02,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:31:02,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:31:02,951 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:31:02,967 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:31:02,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:31:03,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:03,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:03,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:31:03,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:03,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:03,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:31:03,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,367 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:03,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:03,431 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:31:03,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:03,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:31:03,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:31:03,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:31:03,653 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:31:03,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:31:03,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:31:03,714 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:31:03,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:31:03,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:31:03,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:31:03,762 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:31:03,762 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:31:03,762 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:31:03,763 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:31:03,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:31:04,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:31:04,267 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:31:04,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:31:04,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:31:04,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:31:04,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:31:04,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:31:04,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:31:04,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:31:04,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:04,562 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:04,568 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:04,568 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:06,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:06,747 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:06,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:06,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:08,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:08,931 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:08,937 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:08,937 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:10,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:11,114 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:11,119 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:11,119 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:13,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:13,302 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:13,308 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:13,308 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:15,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:15,473 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:15,479 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:15,479 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:17,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:17,651 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:17,658 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:17,658 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:19,660 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:19,832 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:19,838 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:19,838 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:21,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:21,997 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:31:22,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:22,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:24,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:24,236 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:24,236 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:26,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:26,463 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:26,464 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:28,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:28,622 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:28,622 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:30,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:30,806 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:30,808 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:32,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:33,002 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:33,002 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:35,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:35,197 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:35,197 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:37,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:37,398 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:37,400 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:39,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:39,630 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:39,630 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:41,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:41,842 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:41,842 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:43,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:44,055 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:44,055 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:46,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:46,269 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:46,269 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:48,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:48,487 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:48,487 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:50,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:50,634 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:50,634 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:52,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:52,769 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:52,769 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:54,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:55,101 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:55,101 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:57,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:57,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:57,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:31:59,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:31:59,539 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:31:59,539 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:01,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:01,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:01,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:03,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:04,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:04,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:06,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:06,250 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:06,250 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:08,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:08,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:08,497 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:10,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:10,765 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:10,766 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:12,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:12,948 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:12,948 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:14,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:15,187 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:15,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:32:17,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:17,413 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:32:17,414 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 73.6519999504. 2021-12-15 16:32:17,414 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:32:17,417 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:32:17,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:32:17,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:32:17,845 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:32:17,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:32:17,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:32:17,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:32:17,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:32:17,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:32:17,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:32:17,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:32:17,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:18,160 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:18,160 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:32:20,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:20,329 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:20,329 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:32:22,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:22,490 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:22,490 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:32:24,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:24,660 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:24,660 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:32:26,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:26,792 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:32:26,792 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 9.37800002098. 2021-12-15 16:32:26,793 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:32:26,795 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:32:26,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:32:27,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:32:27,296 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:32:27,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:32:27,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:32:27,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:32:27,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:32:27,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:32:27,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:32:27,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:32:27,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:27,654 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:32:27,655 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.862999916077. 2021-12-15 16:32:27,657 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:32:27,657 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:32:27,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:32:27,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:32:27,986 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:32:27,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:32:28,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:32:28,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:32:28,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:32:28,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:32:28,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:32:28,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:32:28,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:28,262 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:28,262 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:32:30,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:30,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:32:30,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 16:32:32,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:32:32,563 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:32:32,563 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 4.90799999237. ############################################################ # # RAM FootPrint: 5446.35546875 # ############################################################ 2021-12-15 16:32:32,598 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:32:32,598 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-15 16:34:02,604 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:34:02,604 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:34:07,605 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:34:07,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:08,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:34:08,263 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:08,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:34:08,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:34:08,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:34:08,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:34:08,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:34:08,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:34:08,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:34:08,440 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:34:08,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:34:08,506 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['97565 memcached\n'], []) 2021-12-15 16:34:08,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:34:08,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:34:08,664 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['97565 memcached\n'], []) 2021-12-15 16:34:08,664 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:34:08,668 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:34:08,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:09,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:34:09,352 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:09,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:34:09,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:34:09,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:34:09,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:34:09,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:34:09,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:34:09,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:34:09,589 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:34:09,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:34:09,674 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['60493 memcached\n'], []) 2021-12-15 16:34:09,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:34:09,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:34:09,848 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['60493 memcached\n'], []) 2021-12-15 16:34:09,848 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:34:09,852 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:34:09,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:10,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:34:10,365 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:10,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:34:10,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:34:10,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:34:10,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:34:10,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:34:10,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:34:10,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:34:10,489 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:34:10,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:34:10,569 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['109565 memcached\n'], []) 2021-12-15 16:34:10,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:34:10,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:34:10,713 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['109565 memcached\n'], []) 2021-12-15 16:34:10,713 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:34:10,716 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:34:10,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:11,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:34:11,207 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:11,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:34:11,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:34:11,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:34:11,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:34:11,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:34:11,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:34:11,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:34:11,378 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:34:11,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:34:11,457 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['141122 memcached\n'], []) 2021-12-15 16:34:11,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:34:11,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:34:11,592 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['141122 memcached\n'], []) 2021-12-15 16:34:11,594 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:34:11,595 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:34:11,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:11,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:34:12,039 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:12,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:34:12,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:34:12,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:34:12,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:34:12,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:34:12,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:34:12,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:34:12,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:34:12,191 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:34:12,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:34:12,242 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:12,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:12,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:12,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:12,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:12,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:12,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:12,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:12,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:12,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:12,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:12,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:34:12,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:34:12,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:12,982 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:13,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:34:13,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:34:13,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:13,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:13,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:34:13,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:34:13,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:34:13,144 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:34:13,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:34:13,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:13,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:34:13,187 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:34:13,187 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:34:13,190 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:34:13,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:13,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:34:13,680 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:13,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:34:13,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:34:13,711 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:34:13,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:34:13,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:34:13,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:34:13,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:34:13,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:34:13,795 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:34:13,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:34:13,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:13,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:13,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:13,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:13,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:14,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:14,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:14,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:14,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:14,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:14,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:14,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:14,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:14,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:14,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:14,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:14,397 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:14,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:34:14,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:34:14,466 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:34:14,480 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:34:14,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:34:14,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:14,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:34:14,519 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:34:14,519 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:34:14,523 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:34:14,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:14,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:34:15,013 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:15,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:34:15,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:34:15,048 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:34:15,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:34:15,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:34:15,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:34:15,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:34:15,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:34:15,140 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:34:15,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:34:15,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:15,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:15,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:15,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:15,326 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:15,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:15,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:15,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:15,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:15,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:15,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:15,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:15,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:34:15,608 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:34:15,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:34:15,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:15,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:34:15,660 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:34:15,660 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:34:15,664 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:34:15,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:17,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:34:17,730 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:17,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:34:17,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:34:17,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:34:17,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:34:17,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:34:17,826 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:34:17,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:34:17,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:34:17,872 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:34:17,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:34:17,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:17,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:18,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:18,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:34:18,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:18,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:18,209 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:18,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:34:18,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:18,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:18,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:18,326 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:34:18,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:18,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:18,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:18,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:34:18,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:18,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:34:18,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:18,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:34:18,482 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:34:18,492 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:34:18,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:34:18,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:34:18,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:34:18,526 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:34:18,528 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:34:18,528 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:34:18,529 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:34:18,635 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:34:18,944 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:34:19,026 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:34:19,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:34:19,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:34:19,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:34:19,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:34:19,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:34:19,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:34:19,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:34:19,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:19,352 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:19,361 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:19,361 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:21,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:21,525 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:21,532 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:21,532 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:23,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:23,703 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:23,709 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:23,709 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:25,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:25,894 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:25,900 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:25,900 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:27,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:28,069 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:28,073 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:28,073 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:30,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:30,180 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:30,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:30,187 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:32,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:32,365 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:32,371 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:32,371 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:34,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:34,558 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:34:34,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:34,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:36,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:36,709 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:36,709 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:38,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:38,914 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:38,915 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:40,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:41,079 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:41,079 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:43,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:43,305 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:43,306 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:45,306 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:45,519 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:45,519 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:47,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:47,742 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:47,742 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:49,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:49,943 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:49,943 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:51,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:52,148 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:52,148 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:54,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:54,359 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:54,359 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:56,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:56,565 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:56,566 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:34:58,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:34:58,772 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:34:58,772 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:00,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:00,979 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:00,980 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:02,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:03,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:03,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:05,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:05,285 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:05,285 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:07,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:07,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:07,496 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:09,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:09,667 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:09,667 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:11,668 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:11,872 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:11,872 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:13,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:14,095 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:14,095 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:16,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:16,358 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:16,359 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:18,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:18,605 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:18,605 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:20,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:20,848 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:20,848 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:22,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:23,075 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:23,075 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:25,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:25,289 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:25,289 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:27,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:27,460 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:27,460 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:35:29,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:29,737 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:35:29,739 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 71.2099997997. 2021-12-15 16:35:29,740 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:35:29,740 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:35:29,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:35:30,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:35:30,154 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:35:30,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:35:30,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:35:30,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:35:30,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:35:30,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:35:30,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:35:30,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:35:30,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:30,426 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:30,427 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:35:32,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:32,602 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:32,602 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:35:34,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:34,783 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:34,783 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:35:36,783 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:36,961 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:36,961 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:35:38,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:39,138 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:35:39,140 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:35:41,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:41,436 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:35:41,436 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.6970000267. 2021-12-15 16:35:41,436 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:35:41,438 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:35:41,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:35:41,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:35:41,953 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:35:41,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:35:41,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:35:41,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:35:42,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:35:42,049 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:35:42,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:35:42,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:35:42,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:42,326 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:35:42,328 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.891999959946. 2021-12-15 16:35:42,328 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:35:42,329 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:35:42,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:35:42,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:35:42,786 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:35:42,788 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:35:42,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:35:42,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:35:42,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:35:42,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:35:42,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:35:42,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:35:42,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:35:43,082 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:35:43,082 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.754999876022. ############################################################ # # RAM FootPrint: 5405.33203125 # ############################################################ 2021-12-15 16:35:43,114 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:35:43,115 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-15 16:37:01,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:37:01,119 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:37:06,121 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:37:06,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:06,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:37:06,828 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:06,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:37:06,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:37:06,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:37:06,882 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:37:06,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:37:06,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:37:06,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:37:06,947 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:37:06,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:37:06,994 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['98869 memcached\n'], []) 2021-12-15 16:37:06,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:37:07,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:37:07,161 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['98869 memcached\n'], []) 2021-12-15 16:37:07,163 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:37:07,165 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:37:07,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:07,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:37:07,727 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:07,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:37:07,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:37:07,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:37:07,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:37:07,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:37:07,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:37:07,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:37:07,888 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:37:07,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:37:07,963 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['61528 memcached\n'], []) 2021-12-15 16:37:07,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:37:08,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:37:08,108 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['61528 memcached\n'], []) 2021-12-15 16:37:08,108 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:37:08,112 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:37:08,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:08,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:37:08,628 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:08,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:37:08,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:37:08,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:37:08,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:37:08,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:37:08,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:37:08,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:37:08,773 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:37:08,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:37:08,858 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['110381 memcached\n'], []) 2021-12-15 16:37:08,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:37:08,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:37:08,997 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['110381 memcached\n'], []) 2021-12-15 16:37:08,999 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:37:09,002 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:37:09,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:09,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:37:09,450 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:09,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:37:09,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:37:09,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:37:09,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:37:09,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:37:09,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:37:09,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:37:09,611 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:37:09,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:37:09,680 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['142259 memcached\n'], []) 2021-12-15 16:37:09,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:37:09,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:37:09,838 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['142259 memcached\n'], []) 2021-12-15 16:37:09,838 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:37:09,842 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:37:09,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:10,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:37:10,217 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:10,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:37:10,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:37:10,260 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:37:10,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:37:10,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:37:10,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:37:10,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:37:10,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:37:10,378 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:37:10,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:37:10,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:10,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:10,522 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:10,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:10,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:10,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:10,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:10,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:10,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:10,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:10,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:10,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:10,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:10,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:11,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:11,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:11,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:11,099 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:11,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:11,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:11,174 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:11,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:11,232 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:37:11,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:37:11,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:11,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:11,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:37:11,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:37:11,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:11,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:11,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:37:11,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:37:11,517 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:37:11,540 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:37:11,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:37:11,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:11,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:37:11,605 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:37:11,607 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:37:11,608 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:37:11,744 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:12,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:37:12,144 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:12,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:37:12,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:37:12,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:37:12,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:37:12,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:37:12,239 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:37:12,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:37:12,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:37:12,286 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:37:12,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:37:12,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:12,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:12,433 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:12,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:12,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:12,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:12,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:12,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:12,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:12,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:12,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:12,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:12,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:12,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:12,849 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:12,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:12,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:12,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:12,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:12,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:13,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:13,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:13,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:37:13,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:37:13,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:37:13,081 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:37:13,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:37:13,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:13,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:37:13,125 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:37:13,125 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:37:13,128 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:37:13,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:13,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:37:13,628 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:13,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:37:13,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:37:13,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:37:13,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:37:13,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:37:13,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:37:13,727 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:37:13,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:37:13,759 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:37:13,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:37:13,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:13,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:13,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:13,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:13,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:13,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:14,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:14,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:14,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:14,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:14,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:14,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:14,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:37:14,168 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:37:14,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:37:14,183 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:14,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:37:14,213 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:37:14,214 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:37:14,217 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:37:14,319 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:14,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:37:14,727 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:14,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:37:14,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:37:14,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:37:14,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:37:14,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:37:14,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:37:14,821 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:37:14,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:37:14,855 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:37:14,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:37:14,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:15,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:15,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:37:15,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:15,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:15,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:37:15,273 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:15,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:15,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:37:15,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:15,477 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:15,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:37:15,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:37:15,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:15,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:37:15,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:37:15,654 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:37:15,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:37:15,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:37:15,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:37:15,700 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:37:15,700 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:37:15,700 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:37:15,703 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:37:15,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:37:17,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:37:17,259 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:37:17,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:37:17,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:37:17,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:37:17,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:37:17,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:37:17,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:37:17,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:37:17,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:17,582 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:17,589 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:17,589 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:19,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:19,746 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:19,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:19,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:21,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:21,923 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:21,927 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:21,927 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:23,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:24,104 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:24,108 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:24,108 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:26,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:26,286 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:26,292 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:26,292 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:28,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:28,473 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:28,479 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:28,479 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:30,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:30,648 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:30,654 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:30,654 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:32,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:32,836 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:32,842 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:32,842 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:34,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:35,015 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:37:35,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:35,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:37,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:37,250 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:37,250 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:39,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:39,433 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:39,433 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:41,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:41,647 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:41,647 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:43,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:43,895 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:43,895 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:45,897 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:46,104 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:46,105 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:48,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:48,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:48,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:50,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:50,548 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:50,549 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:52,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:52,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:52,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:54,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:54,970 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:54,970 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:56,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:57,186 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:57,186 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:37:59,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:37:59,404 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:37:59,404 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:01,404 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:01,607 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:01,607 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:03,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:03,770 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:03,770 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:05,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:05,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:05,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:07,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:08,188 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:08,190 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:10,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:10,424 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:10,424 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:12,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:12,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:12,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:14,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:14,911 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:14,913 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:16,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:17,072 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:17,072 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:19,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:19,316 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:19,316 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:21,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:21,562 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:21,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:23,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:23,769 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:23,769 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:25,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:25,987 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:25,989 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:38:27,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:29,915 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:38:29,917 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 74.2170000076. 2021-12-15 16:38:29,917 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:38:29,918 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:38:29,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:38:30,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:38:30,339 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:38:30,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:38:30,355 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:38:30,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:38:30,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:38:30,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:38:30,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:38:30,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:38:30,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:30,609 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:30,609 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:38:32,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:32,779 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:32,779 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:38:34,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:34,967 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:38:34,967 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:38:36,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:37,200 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:38:37,200 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.28299999237. 2021-12-15 16:38:37,200 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:38:37,203 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:38:37,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:38:37,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:38:37,700 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:38:37,701 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:38:37,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:38:37,747 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:38:37,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:38:37,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:38:37,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:38:37,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:38:37,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:38,062 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:38:38,062 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.862999916077. 2021-12-15 16:38:38,063 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:38:38,065 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:38:38,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:38:38,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:38:38,532 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:38:38,532 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:38:38,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:38:38,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:38:38,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:38:38,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:38:38,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:38:38,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:38:38,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:38:38,862 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:38:38,862 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.799999952316. ############################################################ # # RAM FootPrint: 5326.74609375 # ############################################################ 2021-12-15 16:38:38,894 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:38:38,894 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 83 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-15 16:40:01,898 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:40:01,900 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:40:06,901 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:40:07,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:07,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:40:07,493 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:07,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:40:07,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:40:07,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:40:07,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:40:07,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:40:07,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:40:08,072 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:40:08,094 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:40:08,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:40:08,178 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['100151 memcached\n'], []) 2021-12-15 16:40:08,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:40:08,263 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:40:08,368 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['100151 memcached\n'], []) 2021-12-15 16:40:08,368 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:40:08,371 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:40:08,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:09,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:40:09,526 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:09,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:40:09,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:40:09,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:40:09,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:40:09,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:40:09,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:40:09,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:40:09,637 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:40:09,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:40:09,686 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['62441 memcached\n'], []) 2021-12-15 16:40:09,686 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:40:09,755 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:40:09,884 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['62441 memcached\n'], []) 2021-12-15 16:40:09,884 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:40:09,888 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:40:10,015 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:10,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:40:10,387 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:10,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:40:10,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:40:10,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:40:10,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:40:10,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:40:10,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:40:10,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:40:10,542 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:40:10,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:40:10,618 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['111176 memcached\n'], []) 2021-12-15 16:40:10,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:40:10,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:40:10,743 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['111176 memcached\n'], []) 2021-12-15 16:40:10,743 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:40:10,746 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:40:10,871 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:11,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:40:11,227 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:11,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:40:11,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:40:11,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:40:11,305 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:40:11,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:40:11,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:40:11,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:40:11,407 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:40:11,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:40:11,497 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['143355 memcached\n'], []) 2021-12-15 16:40:11,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:40:11,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:40:11,655 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['143355 memcached\n'], []) 2021-12-15 16:40:11,657 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:40:11,658 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:40:11,765 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:11,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:40:12,061 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:12,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:40:12,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:40:12,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:40:12,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:40:12,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:40:12,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:40:12,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:40:12,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:40:12,221 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:40:12,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:40:12,275 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:12,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:12,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:12,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:12,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:12,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:12,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:12,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:12,637 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:12,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:12,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:12,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:12,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:12,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:12,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:12,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:12,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:12,943 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:12,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:13,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:13,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:13,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:13,102 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:13,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:13,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:13,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:13,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:40:13,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:40:13,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:13,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:13,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:40:13,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:40:13,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:13,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:13,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:40:13,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:40:13,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:40:13,513 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:40:13,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:40:13,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:13,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:40:13,565 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:40:13,565 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:40:13,569 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:40:13,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:13,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:40:14,068 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:14,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:40:14,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:40:14,105 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:40:14,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:40:14,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:40:14,157 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:40:14,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:40:14,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:40:14,203 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:40:14,257 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:40:14,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:14,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:14,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:14,446 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:14,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:14,575 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:14,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:14,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:14,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:14,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:14,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:14,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:14,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:14,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:14,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:14,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:14,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:15,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:15,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:15,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:40:15,055 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:40:15,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:40:15,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:15,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:40:15,104 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:40:15,104 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:40:15,105 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:40:15,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:15,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:40:15,592 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:15,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:40:15,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:40:15,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:40:15,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:40:15,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:40:15,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:40:15,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:40:15,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:40:15,724 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:40:15,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:40:15,782 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:15,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:15,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:15,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:15,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:15,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:15,997 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:16,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:16,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:16,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:16,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:16,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:16,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:16,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:16,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:16,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:16,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:40:16,286 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:40:16,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:40:16,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:16,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:40:16,335 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:40:16,335 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:40:16,338 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:40:16,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:16,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:40:16,845 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:16,846 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:40:16,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:40:16,885 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:40:16,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:40:16,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:40:16,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:40:16,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:40:16,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:40:16,990 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:40:17,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:40:17,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:17,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:40:17,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,295 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:17,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:40:17,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:17,538 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:40:17,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:17,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:40:17,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,726 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:17,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:40:17,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:40:17,841 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:17,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:40:17,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:40:17,888 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:40:17,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:40:17,903 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:40:17,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:40:17,930 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:40:17,930 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:40:17,930 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:40:17,934 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:40:18,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:40:18,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:40:18,421 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:40:18,421 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:40:18,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:40:18,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:40:18,477 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:40:18,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:40:18,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:40:18,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:40:18,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:18,697 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:18,704 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:18,704 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:20,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:20,884 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:20,888 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:20,888 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:22,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:23,076 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:23,082 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:23,082 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:25,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:25,220 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:25,224 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:25,224 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:27,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:27,395 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:27,400 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:27,400 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:29,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:29,572 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:29,579 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:29,579 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:31,579 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:31,750 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:31,756 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:31,756 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:33,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:33,871 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:33,875 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:33,875 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:35,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:36,053 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:40:36,059 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:36,059 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:38,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:38,255 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:38,256 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:40,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:40,489 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:40,490 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:42,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:42,694 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:42,694 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:44,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:44,910 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:44,910 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:46,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:47,138 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:47,138 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:49,140 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:49,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:49,342 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:51,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:51,546 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:51,546 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:53,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:53,747 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:53,747 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:55,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:55,931 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:55,931 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:40:57,933 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:40:58,144 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:40:58,145 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:00,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:00,312 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:00,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:02,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:02,471 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:02,473 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:04,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:04,622 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:04,622 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:06,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:06,789 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:06,789 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:08,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:08,950 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:08,950 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:10,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:11,125 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:11,127 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:13,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:13,298 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:13,298 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:15,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:15,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:15,459 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:17,459 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:17,644 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:17,644 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:19,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:19,826 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:19,826 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:21,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:22,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:22,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:24,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:24,168 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:24,168 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:26,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:26,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:26,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:28,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:28,507 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:28,507 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:30,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:30,673 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:30,673 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:32,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:32,855 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:32,855 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:34,857 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:35,023 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:35,023 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:37,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:37,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:37,204 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:39,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:39,392 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:41:39,392 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:41:41,392 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:41,540 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:41:41,542 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.611000061. 2021-12-15 16:41:41,542 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:41:41,545 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:41:41,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:41:41,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:41:42,068 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:41:42,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:41:42,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:41:42,124 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:41:42,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:41:42,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:41:42,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:41:42,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:41:42,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:42,414 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:41:42,414 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.87299990654. 2021-12-15 16:41:42,414 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:41:42,417 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:41:42,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:41:42,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:41:42,927 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:41:42,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:41:42,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:41:42,966 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:41:42,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:41:43,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:41:43,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:41:43,043 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:41:43,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:43,246 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:41:43,246 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.831000089645. 2021-12-15 16:41:43,246 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:41:43,249 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:41:43,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:41:43,612 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:41:43,696 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:41:43,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:41:43,714 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:41:43,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:41:43,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:41:43,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:41:43,785 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:41:43,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:41:43,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:41:43,979 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:41:43,980 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.733000040054. ############################################################ # # RAM FootPrint: 5315.51953125 # ############################################################ 2021-12-15 16:41:44,013 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:41:44,013 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-15 16:43:35,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:43:35,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:43:40,020 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:43:40,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:40,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:43:40,596 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:40,596 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:43:40,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:43:40,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:43:40,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:43:40,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:43:40,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:43:40,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:43:40,759 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:43:40,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:43:40,815 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['101414 memcached\n'], []) 2021-12-15 16:43:40,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:43:40,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:43:40,980 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['101414 memcached\n'], []) 2021-12-15 16:43:40,980 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:43:40,983 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:43:41,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:42,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:43:42,506 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:42,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:43:42,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:43:42,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:43:42,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:43:42,589 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:43:42,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:43:42,619 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:43:42,634 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:43:42,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:43:42,690 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['63380 memcached\n'], []) 2021-12-15 16:43:42,690 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:43:42,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:43:42,835 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['63380 memcached\n'], []) 2021-12-15 16:43:42,835 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:43:42,839 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:43:42,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:43,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:43:43,359 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:43,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:43:43,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:43:43,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:43:43,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:43:43,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:43:43,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:43:43,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:43:43,509 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:43:43,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:43:43,585 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['112003 memcached\n'], []) 2021-12-15 16:43:43,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:43:43,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:43:43,716 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['112003 memcached\n'], []) 2021-12-15 16:43:43,716 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:43:43,719 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:43:43,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:44,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:43:44,219 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:44,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:43:44,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:43:44,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:43:44,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:43:44,329 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:43:44,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:43:44,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:43:44,398 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:43:44,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:43:44,486 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['144519 memcached\n'], []) 2021-12-15 16:43:44,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:43:44,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:43:44,630 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['144519 memcached\n'], []) 2021-12-15 16:43:44,630 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:43:44,632 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:43:44,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:44,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:43:45,079 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:45,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:43:45,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:43:45,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:43:45,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:43:45,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:43:45,181 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:43:45,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:43:45,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:43:45,233 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:43:45,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:43:45,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:45,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:45,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:45,464 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:45,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:45,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:45,621 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:45,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:45,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:45,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:45,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:45,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:45,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:45,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:45,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:46,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:46,038 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,086 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:46,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:46,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:46,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:46,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:46,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:46,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:46,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:43:46,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:43:46,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:46,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:43:46,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:43:46,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:46,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:43:46,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:43:46,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:43:46,795 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:43:46,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:43:46,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:46,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:43:46,851 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:43:46,851 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:43:46,855 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:43:46,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:47,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:43:47,384 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:47,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:43:47,403 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:43:47,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:43:47,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:43:47,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:43:47,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:43:47,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:43:47,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:43:47,526 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:43:47,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:43:47,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:47,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:47,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:47,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:47,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:47,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:47,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:47,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:47,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:47,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:47,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:47,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:47,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:47,926 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:47,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:47,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:47,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:47,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:48,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:48,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:48,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:48,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:48,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:48,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:48,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:43:48,121 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:43:48,121 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:43:48,132 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:48,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:43:48,157 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:43:48,157 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:43:48,160 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:43:48,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:48,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:43:48,661 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:48,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:43:48,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:43:48,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:43:48,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:43:48,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:43:48,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:43:48,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:43:48,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:43:48,805 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:43:48,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:43:48,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:48,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:48,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:48,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:48,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:49,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:49,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:49,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:49,094 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:49,117 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:49,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:49,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:49,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:49,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:49,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:49,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:49,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:43:49,259 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:43:49,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:43:49,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:49,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:43:49,299 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:43:49,299 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:43:49,302 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:43:49,408 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:49,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:43:49,808 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:49,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:43:49,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:43:49,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:43:49,857 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:43:49,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:43:49,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:43:49,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:43:49,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:43:49,934 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:43:49,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:43:49,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:50,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:43:50,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,186 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:50,265 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:43:50,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:50,368 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:43:50,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,420 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:50,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:43:50,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,551 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:50,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:43:50,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:43:50,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:50,696 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:43:50,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:43:50,723 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:43:50,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:43:50,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:43:50,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:43:50,762 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:43:50,762 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:43:50,762 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:43:50,765 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:43:50,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:43:51,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:43:51,237 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:43:51,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:43:51,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:43:51,272 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:43:51,293 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:43:51,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:43:51,331 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:43:51,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:43:51,365 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:43:51,496 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:43:51,500 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:43:51,502 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:43:53,502 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:43:53,665 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:43:53,671 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:43:53,671 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:43:55,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:43:55,857 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:43:55,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:43:55,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:43:57,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:43:58,033 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:43:58,039 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:43:58,039 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:00,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:00,210 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:44:00,214 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:00,214 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:02,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:02,377 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:44:02,381 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:02,382 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:04,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:04,542 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:44:04,548 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:04,548 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:06,548 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:06,724 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:44:06,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:06,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:08,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:08,894 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:44:08,901 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:08,901 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:10,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:11,039 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:11,039 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:13,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:13,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:13,203 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:15,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:15,417 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:15,417 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:17,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:17,627 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:17,627 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:19,628 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:19,839 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:19,839 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:21,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:21,984 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:21,984 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:23,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:24,171 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:24,171 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:26,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:26,371 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:26,371 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:28,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:28,540 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:28,540 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:30,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:30,753 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:30,753 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:32,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:32,957 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:32,957 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:34,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:35,164 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:35,164 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:37,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:37,323 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:37,323 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:39,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:39,519 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:39,519 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:41,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:41,664 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:41,664 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:43,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:43,828 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:43,829 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:45,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:46,019 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:46,019 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:48,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:48,209 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:48,210 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:50,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:50,388 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:50,390 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:52,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:52,556 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:52,558 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:54,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:54,743 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:54,743 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:56,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:56,915 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:56,917 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:44:58,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:44:59,118 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:44:59,118 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:01,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:01,286 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:01,286 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:03,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:03,457 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:03,457 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:05,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:05,634 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:05,634 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:07,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:07,812 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:07,812 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:09,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:09,996 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:09,996 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:11,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:12,158 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:45:12,158 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:45:14,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:14,371 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:45:14,371 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.6089999676. 2021-12-15 16:45:14,372 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:45:14,374 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:45:14,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:45:14,832 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:45:15,003 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:45:15,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:45:15,028 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:45:15,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:45:15,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:45:15,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:45:15,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:45:15,334 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:45:15,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:15,625 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:45:15,625 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.25300002098. 2021-12-15 16:45:15,628 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:45:15,628 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:45:15,733 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:45:16,029 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:45:16,115 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:45:16,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:45:16,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:45:16,155 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:45:16,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:45:16,197 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:45:16,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:45:16,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:45:16,247 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:16,441 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:45:16,441 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.813999891281. 2021-12-15 16:45:16,443 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:45:16,444 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:45:16,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:45:16,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:45:16,904 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:45:16,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:45:16,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:45:16,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:45:16,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:45:17,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:45:17,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:45:17,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:45:17,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:45:17,286 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:45:17,288 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.84500002861. ############################################################ # # RAM FootPrint: 5300.58984375 # ############################################################ 2021-12-15 16:45:17,322 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:45:17,322 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-15 16:47:09,325 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:47:09,326 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:47:14,328 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:47:14,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:14,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:47:14,911 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:14,913 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:47:14,931 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:47:14,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:47:15,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:47:15,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:47:15,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:47:15,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:47:15,078 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:47:15,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:47:15,163 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['102797 memcached\n'], []) 2021-12-15 16:47:15,163 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:47:15,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:47:15,400 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['102797 memcached\n'], []) 2021-12-15 16:47:15,400 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:47:15,403 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:47:15,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:15,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:47:15,973 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:15,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:47:15,990 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:47:16,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:47:16,023 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:47:16,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:47:16,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:47:16,071 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:47:16,085 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:47:16,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:47:16,125 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['64304 memcached\n'], []) 2021-12-15 16:47:16,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:47:16,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:47:16,256 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['64304 memcached\n'], []) 2021-12-15 16:47:16,257 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:47:16,259 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:47:16,405 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:16,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:47:16,808 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:16,808 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:47:16,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:47:16,877 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:47:16,900 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:47:16,923 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:47:16,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:47:16,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:47:16,980 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:47:16,980 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:47:17,068 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['112886 memcached\n'], []) 2021-12-15 16:47:17,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:47:17,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:47:17,207 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['112886 memcached\n'], []) 2021-12-15 16:47:17,209 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:47:17,210 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:47:17,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:17,617 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:47:17,716 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:17,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:47:17,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:47:17,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:47:17,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:47:17,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:47:17,857 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:47:17,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:47:17,908 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:47:17,908 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:47:17,993 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['145675 memcached\n'], []) 2021-12-15 16:47:17,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:47:18,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:47:18,150 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['145675 memcached\n'], []) 2021-12-15 16:47:18,150 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:47:18,153 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:47:18,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:18,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:47:18,607 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:18,608 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:47:18,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:47:18,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:47:18,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:47:18,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:47:18,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:47:18,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:47:18,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:47:18,790 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:47:18,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:47:18,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:18,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:18,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:18,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:18,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,069 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:19,107 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:19,140 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,173 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,204 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:19,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:19,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:19,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:19,361 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,387 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:19,438 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:19,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:19,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:19,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:47:19,665 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:47:19,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:47:19,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:47:19,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,862 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:19,891 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:47:19,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:47:19,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:47:19,953 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:47:19,953 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:47:19,969 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:19,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:47:20,000 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:47:20,000 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:47:20,003 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:47:20,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:20,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:47:20,526 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:20,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:47:20,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:47:20,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:47:20,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:47:20,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:47:20,622 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:47:20,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:47:20,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:47:20,674 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:47:20,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:47:20,721 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:20,767 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:20,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:20,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:20,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:20,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:20,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:21,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:21,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:21,073 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:21,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:21,135 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:21,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:21,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:21,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:21,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:21,252 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:21,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:21,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:21,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:21,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:21,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:21,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:21,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:21,426 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:47:21,437 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:47:21,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:47:21,448 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:21,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:47:21,474 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:47:21,474 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:47:21,477 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:47:21,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:21,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:47:21,970 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:21,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:47:21,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:47:22,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:47:22,019 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:47:22,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:47:22,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:47:22,066 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:47:22,084 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:47:22,098 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:47:22,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:47:22,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:22,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:22,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:22,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:22,280 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:22,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:22,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:22,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:22,390 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:22,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:22,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:22,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:22,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:22,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:22,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:22,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:22,556 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:47:22,571 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:47:22,571 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:47:22,585 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:22,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:47:22,617 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:47:22,617 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:47:22,619 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:47:22,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:23,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:47:23,112 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:23,112 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:47:23,127 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:47:23,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:47:23,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:47:23,168 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:47:23,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:47:23,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:47:23,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:47:23,219 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:47:23,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:47:23,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,311 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:23,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:47:23,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:23,470 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:47:23,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:23,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:47:23,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:23,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:47:23,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:23,831 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:47:23,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:23,892 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:47:23,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:23,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:47:23,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:47:23,976 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:47:23,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:47:23,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:47:24,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:47:24,026 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:47:24,028 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:47:24,028 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:47:24,029 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:47:24,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:47:24,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:47:24,530 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:47:24,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:47:24,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:47:24,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:47:24,601 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:47:24,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:47:24,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:47:24,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:47:24,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:25,026 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 16:47:25,032 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:25,032 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:27,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:27,198 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:27,203 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:27,203 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:29,203 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:29,369 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:29,375 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:29,375 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:31,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:31,556 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:31,561 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:31,562 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:33,562 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:33,737 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:33,743 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:33,743 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:35,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:35,913 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:35,917 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:35,917 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:37,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:38,082 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:38,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:38,088 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:40,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:40,273 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:40,278 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:40,279 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:42,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:42,447 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:42,451 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:42,453 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:44,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:44,619 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:47:44,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:44,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:46,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:46,790 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:46,790 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:48,792 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:49,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:49,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:51,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:51,204 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:51,206 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:53,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:53,384 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:53,384 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:55,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:55,614 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:55,614 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:57,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:47:57,819 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:47:57,819 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:47:59,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:00,026 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:00,026 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:02,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:02,233 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:02,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:04,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:04,460 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:04,461 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:06,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:06,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:06,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:08,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:08,775 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:08,776 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:10,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:10,921 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:10,921 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:12,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:13,096 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:13,096 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:15,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:15,253 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:15,253 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:17,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:17,413 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:17,413 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:19,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:19,569 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:19,569 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:21,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:21,737 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:21,737 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:23,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:23,904 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:23,905 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:25,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:26,065 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:26,065 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:28,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:28,243 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:28,244 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:30,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:30,417 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:30,417 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:32,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:32,596 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:32,596 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:34,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:34,770 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:34,770 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:36,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:36,950 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:36,950 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:38,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:39,138 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:39,138 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:41,138 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:41,315 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:41,316 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:43,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:43,483 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:43,483 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:45,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:45,661 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:48:45,663 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:48:47,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:48,244 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:48:48,244 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 84.2169997692. 2021-12-15 16:48:48,246 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:48:48,247 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:48:48,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:48:48,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:48:48,859 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:48:48,859 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:48:48,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:48:48,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:48:48,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:48:49,010 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:48:49,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:48:49,055 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:48:49,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:49,276 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:48:49,276 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.03099989891. 2021-12-15 16:48:49,278 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:48:49,279 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:48:49,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:48:49,632 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:48:49,703 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:48:49,704 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:48:49,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:48:49,729 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:48:49,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:48:49,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:48:49,775 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:48:49,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:48:49,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:49,956 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:48:49,956 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.677999973297. 2021-12-15 16:48:49,960 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:48:49,961 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:48:50,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:48:50,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:48:50,438 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:48:50,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:48:50,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:48:50,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:48:50,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:48:50,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:48:50,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:48:50,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:48:50,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:48:50,772 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:48:50,772 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.81200003624. ############################################################ # # RAM FootPrint: 5212.53125 # ############################################################ 2021-12-15 16:48:50,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:48:50,803 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 77 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-15 16:50:07,808 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:50:07,809 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:50:12,809 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:50:12,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:13,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:50:13,479 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:13,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:50:13,496 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:50:13,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:50:13,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:50:13,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:50:13,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:50:13,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:50:13,607 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:50:13,607 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:50:13,667 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['104246 memcached\n'], []) 2021-12-15 16:50:13,667 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:50:13,839 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:50:13,950 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['104246 memcached\n'], []) 2021-12-15 16:50:13,950 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:50:13,954 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:50:14,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:14,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:50:14,555 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:14,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:50:14,576 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:50:14,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:50:14,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:50:14,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:50:14,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:50:14,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:50:14,677 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:50:14,677 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:50:14,736 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['65223 memcached\n'], []) 2021-12-15 16:50:14,736 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:50:14,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:50:14,848 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['65223 memcached\n'], []) 2021-12-15 16:50:14,848 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:50:14,851 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:50:14,961 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:15,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:50:15,338 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:15,339 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:50:15,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:50:15,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:50:15,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:50:15,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:50:15,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:50:15,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:50:15,500 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:50:15,500 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:50:15,582 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['113749 memcached\n'], []) 2021-12-15 16:50:15,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:50:15,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:50:15,723 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['113749 memcached\n'], []) 2021-12-15 16:50:15,723 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:50:15,726 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:50:15,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:16,131 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:50:16,226 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:16,226 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:50:16,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:50:16,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:50:16,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:50:16,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:50:16,348 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:50:16,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:50:16,391 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:50:16,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:50:16,480 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['146848 memcached\n'], []) 2021-12-15 16:50:16,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:50:16,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:50:16,641 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['146848 memcached\n'], []) 2021-12-15 16:50:16,641 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:50:16,644 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:50:16,753 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:17,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:50:17,088 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:17,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:50:17,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:50:17,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:50:17,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:50:17,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:50:17,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:50:17,198 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:50:17,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:50:17,232 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:50:17,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:50:17,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:17,345 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:17,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:17,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:17,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:17,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:17,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:17,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:17,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:17,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:17,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:17,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:17,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:17,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:17,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:17,954 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:17,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:18,081 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:18,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:18,144 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:18,229 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:18,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:18,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:18,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:50:18,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:50:18,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:18,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:50:18,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:50:18,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:18,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:50:18,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:50:18,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:50:18,678 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:50:18,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:50:18,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:18,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:50:18,723 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:50:18,723 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:50:18,726 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:50:18,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:19,141 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:50:19,221 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:19,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:50:19,237 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:50:19,253 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:50:19,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:50:19,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:50:19,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:50:19,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:50:19,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:50:19,345 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:50:19,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:50:19,391 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:19,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:19,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:19,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:19,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:19,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:19,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:19,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:19,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:19,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:19,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:19,743 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:19,770 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:19,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:19,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:19,874 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:19,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:19,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:19,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:19,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:20,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:20,059 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:20,089 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:20,114 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:20,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:20,154 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:20,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:50:20,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:50:20,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:50:20,216 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:50:20,216 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:50:20,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:20,244 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:50:20,259 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:50:20,259 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:50:20,262 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:50:20,380 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:20,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:50:20,750 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:20,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:50:20,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:50:20,780 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:50:20,799 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:50:20,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:50:20,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:50:20,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:50:20,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:50:20,871 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:50:20,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:50:20,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:20,957 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:20,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:21,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:21,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:21,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:21,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:21,148 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:21,176 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:21,211 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:21,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:21,276 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:21,299 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:21,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:21,342 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:21,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:21,372 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:50:21,384 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:50:21,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:50:21,397 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:21,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:50:21,421 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:50:21,421 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:50:21,423 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:50:21,536 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:21,864 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:50:21,940 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:21,940 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:50:21,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:50:21,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:50:21,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:50:22,012 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:50:22,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:50:22,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:50:22,056 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:50:22,072 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:50:22,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:50:22,125 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,167 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,210 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:22,243 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:50:22,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,313 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:22,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:50:22,413 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,451 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,484 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:22,510 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:50:22,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,572 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,599 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:22,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:50:22,650 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,683 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:22,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:50:22,778 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,818 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:50:22,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:22,868 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:50:22,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:50:22,905 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:50:22,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:50:22,921 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:50:22,936 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:50:22,950 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:50:22,950 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:50:22,950 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:50:22,954 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:50:23,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:50:23,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:50:23,473 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:50:23,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:50:23,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:50:23,510 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:50:23,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:50:23,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:50:23,561 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:50:23,578 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:50:23,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:23,844 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:23,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:23,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:25,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:26,039 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:26,045 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:26,045 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:28,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:28,226 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:28,232 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:28,233 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:30,233 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:30,407 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:30,411 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:30,411 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:32,411 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:32,586 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:32,592 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:32,592 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:34,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:34,769 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:34,773 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:34,773 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:36,773 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:36,944 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:36,950 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:36,950 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:38,950 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:39,128 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:39,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:39,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:41,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:41,321 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:50:41,325 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:41,325 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:43,325 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:43,529 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:43,529 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:45,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:45,721 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:45,721 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:47,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:47,875 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:47,875 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:49,875 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:50,107 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:50,107 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:52,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:52,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:52,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:54,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:54,528 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:54,528 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:56,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:56,747 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:56,749 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:50:58,749 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:50:58,888 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:50:58,890 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:00,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:01,088 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:01,088 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:03,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:03,282 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:03,282 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:05,283 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:05,461 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:05,461 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:07,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:07,637 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:07,637 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:09,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:09,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:09,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:11,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:11,963 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:11,963 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:13,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:14,292 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:14,292 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:16,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:16,473 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:16,474 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:18,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:18,642 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:18,642 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:20,642 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:20,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:20,823 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:22,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:23,012 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:23,012 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:25,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:25,190 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:25,191 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:27,191 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:27,377 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:27,377 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:29,377 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:29,558 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:29,558 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:31,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:31,730 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:31,730 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:33,732 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:33,911 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:33,911 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:35,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:36,094 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:36,095 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:38,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:38,276 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:38,278 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:40,278 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:40,453 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:40,454 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:42,454 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:42,625 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:51:42,625 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:51:44,625 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:44,783 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:51:44,783 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 81.8329999447. 2021-12-15 16:51:44,785 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:51:44,786 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:51:44,895 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:51:45,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:51:45,309 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:51:45,309 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:51:45,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:51:45,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:51:45,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:51:45,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:51:45,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:51:45,533 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:51:45,549 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:45,721 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:51:45,723 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.93799996376. 2021-12-15 16:51:45,723 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:51:45,726 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:51:45,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:51:46,142 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:51:46,220 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:51:46,221 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:51:46,246 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:51:46,266 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:51:46,292 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:51:46,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:51:46,335 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:51:46,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:51:46,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:46,576 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:51:46,578 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855000019073. 2021-12-15 16:51:46,578 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:51:46,579 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:51:46,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:51:46,917 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:51:46,986 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:51:46,987 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:51:47,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:51:47,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:51:47,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:51:47,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:51:47,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:51:47,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:51:47,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:51:47,266 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:51:47,266 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.688999891281. ############################################################ # # RAM FootPrint: 5160.7265625 # ############################################################ 2021-12-15 16:51:47,299 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:51:47,299 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-15 16:53:05,303 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:53:05,305 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:53:10,306 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:53:10,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:10,813 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:53:10,894 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:10,894 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:53:10,915 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:53:10,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:53:10,956 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:53:10,976 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:53:10,999 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:53:11,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:53:11,030 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:53:11,030 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:53:11,085 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['105571 memcached\n'], []) 2021-12-15 16:53:11,085 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:53:11,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:53:11,198 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['105571 memcached\n'], []) 2021-12-15 16:53:11,198 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:53:11,201 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:53:11,354 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:11,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:53:11,755 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:11,756 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:53:11,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:53:11,809 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:53:11,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:53:11,890 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:53:11,920 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:53:11,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:53:11,959 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:53:11,959 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:53:12,036 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['66132 memcached\n'], []) 2021-12-15 16:53:12,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:53:12,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:53:12,197 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['66132 memcached\n'], []) 2021-12-15 16:53:12,198 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:53:12,200 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:53:12,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:12,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:53:12,697 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:12,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:53:12,720 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:53:12,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:53:12,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:53:12,796 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:53:12,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:53:12,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:53:12,852 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:53:12,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:53:12,924 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['114577 memcached\n'], []) 2021-12-15 16:53:12,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:53:13,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:53:13,071 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['114577 memcached\n'], []) 2021-12-15 16:53:13,072 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:53:13,073 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:53:13,193 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:13,460 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:53:13,543 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:13,545 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:53:13,569 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:53:13,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:53:13,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:53:13,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:53:13,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:53:13,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:53:13,713 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:53:13,713 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:53:13,803 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['148004 memcached\n'], []) 2021-12-15 16:53:13,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:53:13,884 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:53:13,950 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['148004 memcached\n'], []) 2021-12-15 16:53:13,950 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:53:13,953 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:53:14,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:14,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:53:14,394 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:14,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:53:14,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:53:14,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:53:14,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:53:14,474 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:53:14,492 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:53:14,507 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:53:14,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:53:14,542 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:53:14,586 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:53:14,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:14,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:14,684 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:14,717 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:14,750 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:14,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:14,834 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:14,869 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:14,905 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:14,947 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:14,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:15,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:15,053 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,088 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:15,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:15,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,207 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:15,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:15,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,341 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:15,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:15,444 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:15,530 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:15,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,594 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:53:15,664 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:53:15,691 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,730 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:15,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:53:15,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:53:15,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:53:15,828 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:53:15,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:53:15,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:15,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:53:15,875 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:53:15,875 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:53:15,878 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:53:16,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:16,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:53:16,394 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:16,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:53:16,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:53:16,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:53:16,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:53:16,469 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:53:16,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:53:16,499 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:53:16,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:53:16,532 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:53:16,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:53:16,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:16,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:16,707 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:16,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:16,772 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:16,828 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:16,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:16,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:16,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:16,979 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:17,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:17,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:17,075 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:17,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:17,158 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:17,190 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:17,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:17,256 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:17,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:17,322 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:17,346 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:17,385 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:17,427 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:17,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:17,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:17,494 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:17,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:17,535 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:17,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:53:17,566 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:53:17,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:53:17,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:17,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:53:17,614 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:53:17,614 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:53:17,617 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:53:17,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:18,020 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:53:18,096 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:18,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:53:18,111 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:53:18,130 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:53:18,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:53:18,164 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:53:18,177 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:53:18,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:53:18,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:53:18,213 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:53:18,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:53:18,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:18,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:18,316 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:18,338 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:18,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:18,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:18,410 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:18,428 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:18,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:18,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:18,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:18,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:18,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:18,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:18,588 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:18,604 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:18,618 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:53:18,628 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:53:18,630 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:53:18,641 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:18,654 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:53:18,667 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:53:18,667 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:53:18,671 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:53:18,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:19,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:53:19,180 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:19,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:53:19,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:53:19,217 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:53:19,236 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:53:19,255 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:53:19,269 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:53:19,282 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:53:19,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:53:19,311 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:53:19,336 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:53:19,358 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,401 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:19,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:53:19,519 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,609 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:19,645 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:53:19,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,700 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:19,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:53:19,766 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,790 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,812 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:19,829 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:53:19,845 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,887 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:19,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:53:19,914 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:19,938 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:19,960 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:19,974 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:53:19,989 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:20,002 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:53:20,013 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:20,025 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:53:20,035 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:53:20,046 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:53:20,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:53:20,061 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:53:20,076 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:53:20,089 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:53:20,089 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:53:20,091 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:53:20,095 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:53:20,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:53:20,520 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:53:20,598 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:53:20,598 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:53:20,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:53:20,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:53:20,648 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:53:20,663 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:53:20,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:53:20,687 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:53:20,697 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:20,821 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:20,825 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:20,825 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:22,825 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:22,979 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:22,984 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:22,984 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:24,984 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:25,164 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:25,170 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:25,170 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:27,170 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:27,328 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:27,332 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:27,332 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:29,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:29,497 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:29,503 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:29,503 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:31,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:31,687 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:31,691 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:31,691 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:33,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:33,855 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:33,861 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:33,861 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:35,861 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:36,032 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:36,038 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:36,038 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:38,039 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:38,209 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:38,213 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:38,214 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:40,214 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:40,381 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:53:40,387 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:40,387 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:42,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:42,612 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:42,612 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:44,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:44,822 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:44,822 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:46,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:47,003 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:47,003 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:49,003 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:49,213 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:49,213 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:51,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:51,413 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:51,414 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:53,414 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:53,592 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:53,592 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:55,592 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:55,750 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:55,752 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:57,752 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:53:57,911 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:53:57,911 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:53:59,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:00,062 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:00,063 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:02,063 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:02,229 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:02,230 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:04,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:04,388 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:04,388 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:06,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:06,538 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:06,539 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:08,539 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:08,697 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:08,698 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:10,698 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:10,851 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:10,851 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:12,851 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:13,000 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:13,000 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:15,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:15,161 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:15,161 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:17,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:17,311 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:17,312 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:19,312 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:19,471 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:19,473 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:21,473 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:21,634 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:21,634 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:23,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:23,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:23,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:25,803 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:25,971 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:25,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:27,973 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:28,132 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:28,132 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:30,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:30,315 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:30,315 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:32,315 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:32,496 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:32,497 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:34,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:34,670 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:34,670 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:36,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:36,854 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:36,854 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:38,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:39,030 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:39,030 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:41,032 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:41,211 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:54:41,213 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:54:43,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:43,411 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:54:43,413 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.3220000267. 2021-12-15 16:54:43,414 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:54:43,415 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:54:43,565 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:54:49,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:54:49,944 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:54:49,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:54:49,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:54:49,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:54:50,007 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:54:50,062 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:54:50,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:54:50,095 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:54:50,115 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:50,316 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:54:50,318 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 6.90400004387. 2021-12-15 16:54:50,319 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:54:50,321 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:54:50,424 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:54:50,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:54:50,813 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:54:50,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:54:50,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:54:50,855 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:54:50,881 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:54:50,907 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:54:50,927 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:54:50,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:54:50,970 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:51,190 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:54:51,190 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871999979019. 2021-12-15 16:54:51,190 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:54:51,194 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:54:51,298 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:54:51,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:54:51,638 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:54:51,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:54:51,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:54:51,680 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:54:51,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:54:51,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:54:51,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:54:51,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:54:51,776 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:54:51,937 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:54:51,937 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.746999979019. ############################################################ # # RAM FootPrint: 5143.0 # ############################################################ 2021-12-15 16:54:51,967 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 16:54:51,969 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-15 16:55:57,973 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 16:55:57,973 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 16:56:02,976 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:56:03,109 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:03,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:56:03,460 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:03,461 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:56:03,479 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:56:03,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:56:03,510 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:56:03,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:56:03,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:56:03,552 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:56:03,565 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:56:03,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:56:03,624 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['106898 memcached\n'], []) 2021-12-15 16:56:03,624 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 16:56:03,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 16:56:03,746 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['106898 memcached\n'], []) 2021-12-15 16:56:03,746 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:56:03,750 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:56:03,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:04,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:56:04,384 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:04,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:56:04,400 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:56:04,417 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:56:04,434 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:56:04,453 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:56:04,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:56:04,526 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:56:04,542 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:56:04,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:56:04,627 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['67051 memcached\n'], []) 2021-12-15 16:56:04,627 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 16:56:04,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 16:56:04,846 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['67051 memcached\n'], []) 2021-12-15 16:56:04,848 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:56:04,851 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:56:04,993 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:05,288 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:56:05,374 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:05,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:56:05,394 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:56:05,415 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:56:05,440 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:56:05,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:56:05,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:56:05,506 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:56:05,529 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:56:05,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:56:05,602 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['115399 memcached\n'], []) 2021-12-15 16:56:05,602 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-15 16:56:05,658 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-15 16:56:05,703 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['115399 memcached\n'], []) 2021-12-15 16:56:05,703 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:56:05,706 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:56:05,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:06,058 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:56:06,150 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:06,150 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:56:06,171 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:56:06,194 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:56:06,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:56:06,250 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:56:06,270 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:56:06,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:56:06,308 | infra | DEBUG | Thread-1 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 16:56:06,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:56:06,374 | infra | DEBUG | Thread-1 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['149124 memcached\n'], []) 2021-12-15 16:56:06,374 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 16:56:06,456 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 16:56:06,529 | infra | DEBUG | Thread-1 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['149124 memcached\n'], []) 2021-12-15 16:56:06,529 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:56:06,532 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:56:06,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:06,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:56:07,003 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:07,005 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:56:07,026 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:56:07,045 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:56:07,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:56:07,086 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:56:07,104 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:56:07,119 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:56:07,137 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:56:07,154 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 16:56:07,180 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:56:07,206 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,249 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,290 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:07,323 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:07,352 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,395 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:07,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:07,503 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,558 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,605 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:07,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:07,676 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:07,769 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:07,793 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,822 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,848 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:07,872 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:07,898 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:07,941 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:07,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:08,016 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:08,046 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:08,092 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:08,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:08,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:08,188 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:08,227 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:08,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:56:08,285 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 16:56:08,308 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:08,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:08,381 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:56:08,407 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 16:56:08,430 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:08,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:08,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 16:56:08,483 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 16:56:08,497 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:56:08,513 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 16:56:08,513 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:56:08,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:08,543 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:56:08,559 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 16:56:08,559 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:56:08,562 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:56:08,670 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:08,977 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:56:09,051 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:09,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:56:09,065 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:56:09,079 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:56:09,096 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:56:09,118 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:56:09,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:56:09,145 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:56:09,160 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:56:09,174 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 16:56:09,200 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:56:09,223 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,262 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,301 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:09,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:09,351 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,388 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,423 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:09,457 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:09,487 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,515 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,542 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:09,568 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:09,591 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,614 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,644 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:09,671 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:09,693 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:09,789 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:09,819 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,865 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:09,904 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:09,937 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:09,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:09,994 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:10,017 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:10,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:10,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:56:10,068 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 16:56:10,068 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:56:10,082 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:10,098 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:56:10,111 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 16:56:10,111 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:56:10,114 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:56:10,230 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:10,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:56:10,615 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:10,615 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:56:10,634 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:56:10,651 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:56:10,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:56:10,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:56:10,710 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:56:10,724 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:56:10,746 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:56:10,762 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 16:56:10,795 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:56:10,816 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:10,858 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:10,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:10,934 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:10,963 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:11,000 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:11,042 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:11,078 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:11,108 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:11,147 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:11,184 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:11,213 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:11,240 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:11,267 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:11,296 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:11,321 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:11,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:56:11,359 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 16:56:11,359 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:56:11,378 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:11,398 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:56:11,413 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 16:56:11,413 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:56:11,414 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:56:11,523 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:11,842 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:56:11,930 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:11,930 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:56:11,948 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:56:11,967 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:56:11,986 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:56:12,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:56:12,022 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:56:12,036 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:56:12,052 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 16:56:12,066 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 16:56:12,128 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 16:56:12,151 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,187 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,224 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:12,259 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 16:56:12,289 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,332 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,382 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:12,418 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 16:56:12,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,486 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:12,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 16:56:12,581 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,611 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,640 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:12,661 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 16:56:12,681 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,703 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,723 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:12,742 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 16:56:12,759 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,786 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,806 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:12,823 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 16:56:12,838 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,852 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 16:56:12,867 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:12,878 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 16:56:12,888 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 16:56:12,900 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 16:56:12,901 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 16:56:12,911 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 16:56:12,924 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 16:56:12,934 | test | DEBUG | Thread-1 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 16:56:12,934 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:56:12,936 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:56:12,937 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 16:56:13,051 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:56:13,362 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 16:56:13,443 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:56:13,443 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 16:56:13,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 16:56:13,480 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 16:56:13,505 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 16:56:13,528 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 16:56:13,546 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 16:56:13,563 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 16:56:13,584 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:13,736 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:13,740 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:13,740 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:15,740 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:15,913 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:15,917 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:15,917 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:17,918 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:18,095 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:18,101 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:18,101 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:20,101 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:20,273 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:20,279 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:20,279 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:22,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:22,463 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:22,467 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:22,467 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:24,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:24,638 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:24,647 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:24,647 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:26,647 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:26,829 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:26,834 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:26,834 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:28,835 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:29,002 | test | WARNING | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 16:56:29,006 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:29,006 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:31,006 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:31,165 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:31,165 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:33,165 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:33,369 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:33,369 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:35,369 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:35,582 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:35,582 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:37,582 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:37,801 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:37,802 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:39,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:40,007 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:40,007 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:42,009 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:42,219 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:42,219 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:44,219 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:44,434 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:44,436 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:46,436 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:46,595 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:46,595 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:48,595 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:48,802 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:48,803 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:50,805 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:50,994 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:50,994 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:52,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:53,191 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:53,201 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:55,201 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:55,342 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:55,344 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:57,344 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:57,490 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:57,490 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:56:59,490 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:56:59,637 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:56:59,638 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:01,638 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:01,796 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:01,796 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:03,798 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:03,963 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:03,964 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:05,964 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:06,134 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:06,134 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:08,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:08,302 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:08,302 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:10,302 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:10,470 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:10,470 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:12,471 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:12,631 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:12,631 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:14,631 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:14,813 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:14,813 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:16,815 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:16,983 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:16,983 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:18,983 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:19,161 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:19,161 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:21,161 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:21,328 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:21,328 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:23,328 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:23,489 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:23,489 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:25,489 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:25,677 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:25,678 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 16:57:27,678 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:27,888 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 16:57:27,888 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 74.9529998302. 2021-12-15 16:57:27,891 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:57:27,892 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 16:57:27,996 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:57:28,286 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 16:57:28,371 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:57:28,371 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 16:57:28,384 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 16:57:28,437 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 16:57:28,450 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 16:57:28,467 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 16:57:28,516 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 16:57:28,529 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 16:57:28,559 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:28,763 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:28,763 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:57:30,763 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:30,944 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 16:57:30,944 | test | INFO | Thread-1 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 16:57:32,946 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:33,151 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 16:57:33,151 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.26300001144. 2021-12-15 16:57:33,153 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:57:33,154 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 16:57:33,279 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:57:33,566 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 16:57:33,655 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:57:33,657 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 16:57:33,674 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 16:57:33,694 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 16:57:33,716 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 16:57:33,739 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 16:57:33,757 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 16:57:33,779 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 16:57:33,802 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:34,019 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 16:57:34,019 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.868000030518. 2021-12-15 16:57:34,020 | test | DEBUG | Thread-1 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 16:57:34,023 | infra | DEBUG | Thread-1 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 16:57:34,134 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 16:57:34,375 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 16:57:34,447 | infra | DEBUG | Thread-1 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 16:57:34,447 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 16:57:34,463 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 16:57:34,476 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 16:57:34,493 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 16:57:34,509 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 16:57:34,525 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 16:57:34,540 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 16:57:34,555 | infra | DEBUG | Thread-1 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 16:57:34,717 | infra | DEBUG | Thread-1 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 16:57:34,717 | test | INFO | Thread-1 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.697000026703. 2021-12-15 16:57:34,721 | test | DEBUG | Thread-1 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-15 16:57:34,721 | test | DEBUG | Thread-1 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 6: Rebalance Out with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66', '172.23.110.68'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67'] Servers coming in : [] with services: ['kv'] Servers going out : ['172.23.110.68'] NEW AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.68'] 2021-12-15 17:02:34,757 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639616554.72 2021-12-15 17:02:34,759 | infra | INFO | pool-3-thread-25 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639616554.72' started 2021-12-15 17:02:34,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L493 2021-12-15 17:02:34,769 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-15 17:02:34,783 | test | DEBUG | pool-3-thread-25 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-15 17:02:34,783 | test | DEBUG | pool-3-thread-25 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-15 17:02:34,838 | test | DEBUG | pool-3-thread-25 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 17:02:34,844 | test | INFO | pool-3-thread-25 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 21.7361403509 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 18.3995400816 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 21.6087730861 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 18.855463831 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 17:02:34,852 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:02:34,852 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:02:34,868 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:02:34,869 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:02:34,869 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:02:44,881 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:02:44,882 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:02:44,898 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:02:44,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:02:44,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:02:54,927 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:02:54,927 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:02:54,947 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:02:54,950 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:02:54,950 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:04,963 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:04,963 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:04,980 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:04,983 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:04,983 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:14,996 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:14,996 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:15,013 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:15,013 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:15,015 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:25,026 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:25,028 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:25,043 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:25,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:25,048 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:34,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-15 17:03:34,773 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-15 17:03:35,059 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:35,061 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:35,075 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:35,078 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:35,078 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:45,091 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:45,091 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:45,105 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:45,108 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:45,108 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:03:55,119 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:03:55,119 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:03:55,134 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:03:55,138 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:03:55,138 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:04,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:04,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:04,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:05,148 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:05,150 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:04:05,164 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:05,167 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:05,167 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:14,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:14,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:14,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:15,177 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:15,177 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:04:15,190 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:15,194 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:15,194 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:24,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:24,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:24,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:25,204 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:04:25,204 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:04:25,220 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:25,223 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:25,224 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:34,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:34,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:34,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:35,236 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:35,236 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:04:35,253 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:35,256 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:35,256 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:44,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:44,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:44,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:45,266 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:45,266 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:04:45,282 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:45,286 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:45,286 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:54,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:54,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:04:54,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:04:55,296 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:04:55,298 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:04:55,312 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:04:55,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:04:55,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:04,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:04,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:04,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:05,328 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:05,328 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:05,345 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:05,348 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:05,348 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:14,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:14,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:14,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:15,358 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:15,359 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:15,375 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:15,378 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:15,378 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:25,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:25,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:25,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:25,391 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:25,391 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:25,405 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:25,408 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:25,408 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:35,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:35,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:35,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:35,421 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:35,421 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:35,437 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:35,440 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:35,440 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:45,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:45,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:45,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:45,453 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:45,453 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:45,470 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:45,473 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:45,473 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:55,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:55,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:05:55,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:05:55,489 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:05:55,489 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:05:55,506 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:05:55,509 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:05:55,509 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:05,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:06:05,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:05,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:05,519 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:06:05,519 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:06:05,536 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:05,539 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:05,539 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:15,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:06:15,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:15,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:15,549 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-15 17:06:15,551 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.2 2021-12-15 17:06:15,568 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:15,571 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:15,571 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:25,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:25,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:25,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:25,585 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:25,585 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:06:25,602 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:25,605 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:25,605 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:35,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:35,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:35,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:35,615 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:35,615 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:06:35,632 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:35,635 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:35,635 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:45,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:45,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:45,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:45,647 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:45,647 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:06:45,663 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:45,664 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:45,664 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:55,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:55,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:06:55,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:06:55,676 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:06:55,677 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:06:55,693 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:06:55,696 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:06:55,696 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:05,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:05,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:05,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:05,707 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:05,707 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:05,723 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:05,724 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:05,726 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:15,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:15,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:15,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:15,736 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:15,736 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:15,750 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:15,753 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:15,753 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:25,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:25,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:25,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:25,763 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:25,763 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:25,779 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:25,780 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:25,780 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:35,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:35,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:35,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:35,792 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:35,792 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:35,809 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:35,811 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:35,811 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:45,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:45,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:45,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:45,822 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:45,822 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:45,838 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:45,842 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:45,842 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:55,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:55,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:07:55,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:07:55,851 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:07:55,852 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:07:55,868 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:07:55,871 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:07:55,871 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:05,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:05,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:05,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:05,882 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:05,882 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:05,900 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:05,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:05,901 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:15,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:15,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:15,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:15,913 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:15,913 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:15,928 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:15,931 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:15,931 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:25,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:25,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:25,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:25,943 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:25,944 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:25,960 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:25,963 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:25,963 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:35,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:35,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:35,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:35,976 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:35,976 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:35,993 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:35,996 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:35,996 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:45,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:45,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:45,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:46,010 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:46,010 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:46,032 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:46,035 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:46,036 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:55,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:55,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:08:55,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:08:56,046 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:08:56,048 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:08:56,065 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:08:56,068 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:08:56,069 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:05,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:05,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:05,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:06,081 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:06,081 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:06,101 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:06,105 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:06,107 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:15,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:15,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:15,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:16,119 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:16,119 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:16,141 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:16,144 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:16,144 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:25,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:25,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:25,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:26,158 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:26,160 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:26,181 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:26,184 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:26,184 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:35,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:35,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:35,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:36,197 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:36,197 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:36,223 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:36,224 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:36,226 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:45,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:45,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:45,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:46,236 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:46,237 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:46,255 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:46,257 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:46,257 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:55,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:55,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:09:55,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:09:56,272 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:09:56,273 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:09:56,290 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:09:56,293 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:09:56,293 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:05,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:05,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:05,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:06,309 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:06,309 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:10:06,331 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:06,335 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:06,336 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:15,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:15,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:15,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:16,351 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:16,352 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:10:16,371 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:16,374 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:16,375 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:25,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:25,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:25,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:26,388 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:26,388 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:10:26,408 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:26,411 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:26,413 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:35,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:35,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:35,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:36,427 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:36,427 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:10:36,447 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:36,450 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:36,451 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:45,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:45,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:45,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:46,464 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:46,464 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.39 2021-12-15 17:10:46,483 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:46,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:46,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:55,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-15 17:10:55,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:10:55,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:10:56,497 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:10:56,499 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:10:56,516 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:10:56,519 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:10:56,519 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:05,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:05,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:05,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:06,530 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:06,530 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:06,549 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:06,551 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:06,552 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:15,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:15,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:15,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:16,562 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:16,562 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:16,578 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:16,582 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:16,582 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:25,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:25,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:25,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:26,592 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:26,592 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:26,609 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:26,612 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:26,612 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:35,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:35,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:35,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:36,624 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:36,624 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:36,640 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:36,641 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:36,641 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:45,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:45,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:45,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:46,654 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:46,654 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:46,671 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:46,674 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:46,674 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:55,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:55,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:11:55,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:11:56,686 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:11:56,686 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:11:56,703 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:11:56,704 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:11:56,706 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:05,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:12:05,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:05,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:06,716 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:12:06,716 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:12:06,730 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:06,733 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:06,733 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:15,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:12:15,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:15,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:16,743 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-15 17:12:16,744 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.59 2021-12-15 17:12:16,760 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:16,763 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:16,763 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:25,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:25,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:25,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:26,775 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:26,776 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:12:26,792 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:26,795 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:26,795 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:35,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:35,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:35,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:36,806 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:36,806 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:12:36,823 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:36,825 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:36,825 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:45,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:45,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:45,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:46,835 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:46,836 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:12:46,851 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:46,854 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:46,854 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:56,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:56,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:12:56,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:12:56,865 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:12:56,865 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:12:56,881 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:12:56,884 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:12:56,884 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:06,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:06,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:06,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:06,898 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:06,898 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:13:06,915 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:06,917 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:06,917 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:16,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:16,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:16,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:16,927 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:16,928 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:13:16,947 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:16,950 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:16,950 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:26,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:26,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:26,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:26,960 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-15 17:13:26,960 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.78 2021-12-15 17:13:26,979 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:26,980 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:26,982 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:36,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.98 % 2021-12-15 17:13:36,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:36,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:36,990 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.98 % 2021-12-15 17:13:36,990 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 0.98 2021-12-15 17:13:37,007 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:37,010 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:37,010 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:46,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:13:46,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:46,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:47,023 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:13:47,025 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 17:13:47,043 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:47,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:47,046 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:56,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:13:56,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:13:56,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:13:57,059 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:13:57,059 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 17:13:57,078 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:13:57,082 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:13:57,082 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:06,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:06,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:06,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:07,092 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:07,092 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 17:14:07,111 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:07,115 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:07,115 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:16,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:16,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:16,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:17,128 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:17,128 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 17:14:17,144 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:17,147 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:17,148 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:26,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:26,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:26,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:27,160 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-15 17:14:27,160 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.17 2021-12-15 17:14:27,180 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:27,184 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:27,184 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:36,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:36,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:36,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:37,194 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:37,194 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:14:37,210 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:37,213 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:37,213 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:46,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:46,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:46,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:47,223 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:47,223 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:14:47,240 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:47,243 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:47,243 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:56,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:56,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:14:56,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:14:57,255 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:14:57,255 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:14:57,275 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:14:57,278 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:14:57,278 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:06,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:06,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:06,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:07,289 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:07,289 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:15:07,305 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:07,308 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:07,309 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:16,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:16,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:17,319 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:17,319 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:15:17,336 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:17,338 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:17,338 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:26,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:26,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:26,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:27,351 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:27,351 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 17:15:27,369 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:27,372 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:27,372 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:36,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 17:15:36,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:36,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:37,384 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:15:37,385 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 17:15:37,405 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:37,408 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:37,410 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:46,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:15:46,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:46,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:47,420 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:15:47,421 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 17:15:47,437 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:47,438 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:47,440 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:56,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:15:56,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:15:56,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:15:57,450 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:15:57,450 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 17:15:57,470 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:15:57,473 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:15:57,473 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:06,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:16:06,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:06,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:07,483 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-15 17:16:07,484 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.56 2021-12-15 17:16:07,500 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:07,503 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:07,503 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:16,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:16,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:16,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:17,513 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:17,515 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:16:17,530 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:17,536 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:17,536 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:26,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:26,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:26,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:27,549 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:27,551 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:16:27,566 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:27,569 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:27,569 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:36,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:36,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:36,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:37,582 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:37,582 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:16:37,598 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:37,601 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:37,601 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:46,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:46,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:46,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:47,614 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:47,614 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:16:47,630 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:47,631 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:47,632 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:56,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:56,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:16:56,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:16:57,641 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:16:57,641 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:16:57,658 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:16:57,661 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:16:57,661 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:06,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:17:06,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:06,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:07,677 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.95 % 2021-12-15 17:17:07,677 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 1.95 2021-12-15 17:17:07,696 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:07,698 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:07,698 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:16,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:16,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:16,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:17,710 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:17,710 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:17:17,727 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:17,730 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:17,730 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:26,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:26,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:26,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:27,743 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:27,743 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:17:27,763 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:27,766 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:27,766 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:36,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:36,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:36,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:37,776 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:37,776 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:17:37,793 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:37,796 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:37,796 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:46,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:46,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:46,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:47,808 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:47,808 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:17:47,825 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:47,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:47,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:56,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:56,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:17:56,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:17:57,838 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:17:57,838 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:17:57,855 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:17:57,858 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:17:57,858 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:06,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:18:06,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:06,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:07,868 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:18:07,868 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:18:07,885 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:07,888 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:07,890 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:16,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:18:16,683 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:16,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:17,900 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-15 17:18:17,900 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.34 2021-12-15 17:18:17,918 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:17,937 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:17,937 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:26,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 17:18:26,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:26,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:27,947 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-15 17:18:27,947 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 2.93 2021-12-15 17:18:27,963 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:27,966 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:27,966 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:36,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:36,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:36,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:37,976 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:37,977 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:18:37,999 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:38,003 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:38,003 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:46,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:46,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:46,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:48,013 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:48,013 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:18:48,032 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:48,033 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:48,033 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:56,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:56,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:18:56,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:18:58,045 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:18:58,045 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:18:58,062 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:18:58,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:18:58,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:06,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:06,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:06,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:08,076 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:08,078 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:19:08,095 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:08,098 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:08,098 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:16,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:16,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:16,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:18,108 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:18,109 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:19:18,127 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:18,130 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:18,130 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:26,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:26,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:26,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:28,140 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:28,141 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.13 2021-12-15 17:19:28,157 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:28,160 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:28,160 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:36,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.13 % 2021-12-15 17:19:36,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:36,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:38,170 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-15 17:19:38,170 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.32 2021-12-15 17:19:38,186 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:38,188 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:38,188 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:46,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:19:46,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:46,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:48,200 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:19:48,200 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:19:48,217 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:48,220 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:48,220 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:56,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:19:56,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:19:56,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:19:58,230 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:19:58,230 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:19:58,250 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:19:58,253 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:19:58,253 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:06,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:06,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:06,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:08,265 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:08,265 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:20:08,280 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:08,282 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:08,282 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:16,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:16,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:16,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:18,292 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:18,292 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:20:18,309 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:18,311 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:18,311 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:26,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:26,936 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:26,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:28,321 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:20:28,321 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:20:28,335 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:28,338 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:28,338 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:36,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 17:20:36,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:36,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:38,352 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 17:20:38,354 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 17:20:38,372 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:38,375 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:38,375 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:46,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 17:20:46,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:46,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:48,385 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 17:20:48,385 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 17:20:48,401 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:48,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:48,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:56,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 17:20:56,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:20:56,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:20:58,414 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-15 17:20:58,414 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-15 17:20:58,430 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:20:58,433 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:20:58,433 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:07,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:07,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:07,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:08,443 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:08,444 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:08,460 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:08,461 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:08,463 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:17,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:17,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:17,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:18,473 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:18,473 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:18,490 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:18,492 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:18,492 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:27,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:27,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:27,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:28,503 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:28,503 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:28,519 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:28,523 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:28,523 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:37,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:37,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:37,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:38,536 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:38,536 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:38,555 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:38,559 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:38,559 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:47,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:47,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:47,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:48,571 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:48,572 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:48,588 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:48,592 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:48,592 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:57,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:57,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:21:57,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:21:58,601 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:21:58,602 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:21:58,618 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:21:58,621 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:21:58,621 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:07,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:22:07,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:07,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:08,634 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-15 17:22:08,634 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-15 17:22:08,653 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:08,655 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:08,655 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:17,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.50 % 2021-12-15 17:22:17,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:17,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:18,674 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.50 % 2021-12-15 17:22:18,674 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.5 2021-12-15 17:22:18,693 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:18,696 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:18,697 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:27,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.50 % 2021-12-15 17:22:27,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:27,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:28,710 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.50 % 2021-12-15 17:22:28,710 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.5 2021-12-15 17:22:28,733 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:28,737 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:28,737 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:37,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:37,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:37,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:38,749 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:38,749 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-15 17:22:38,766 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:38,767 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:38,767 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:47,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:47,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:47,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:48,779 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:48,779 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-15 17:22:48,798 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:48,801 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:48,801 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:57,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:57,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:22:57,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:22:58,813 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-15 17:22:58,815 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-15 17:22:58,831 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:22:58,832 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:22:58,832 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:07,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:07,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:07,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:08,844 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:08,844 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 17:23:08,861 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:08,864 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:08,864 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:17,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:17,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:17,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:18,875 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:18,875 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 17:23:18,890 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:18,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:18,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:27,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:27,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:27,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:28,903 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:28,903 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 17:23:28,918 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:28,921 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:28,921 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:37,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:37,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:37,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:38,930 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:38,931 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 17:23:38,947 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:38,948 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:38,948 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:47,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:47,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:47,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:48,959 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 17:23:48,959 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 17:23:48,976 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:48,977 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:48,977 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:57,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:23:57,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:23:57,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:23:58,987 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:23:58,987 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.28 2021-12-15 17:23:59,005 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:23:59,006 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:23:59,006 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:07,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:07,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:07,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:09,016 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:09,017 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.28 2021-12-15 17:24:09,033 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:09,036 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:09,036 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:17,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:17,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:17,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:19,046 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:19,046 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.28 2021-12-15 17:24:19,062 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:19,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:19,065 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:27,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:27,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:27,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:29,075 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:29,075 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.28 2021-12-15 17:24:29,092 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:29,095 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:29,095 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:37,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.28 % 2021-12-15 17:24:37,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:37,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:39,108 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:24:39,108 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.47 2021-12-15 17:24:39,130 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:39,134 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:39,134 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:47,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:24:47,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:47,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:49,148 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:24:49,148 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.47 2021-12-15 17:24:49,165 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:49,170 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:49,170 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:57,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:24:57,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:24:57,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:24:59,181 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:24:59,181 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.47 2021-12-15 17:24:59,198 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:24:59,200 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:24:59,200 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:07,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:25:07,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:07,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:09,217 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.47 % 2021-12-15 17:25:09,217 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.47 2021-12-15 17:25:09,236 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:09,240 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:09,240 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:17,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.86 % 2021-12-15 17:25:17,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:17,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:19,253 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.86 % 2021-12-15 17:25:19,253 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 5.86 2021-12-15 17:25:19,269 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:19,272 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:19,272 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:27,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:27,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:27,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:29,282 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:29,282 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 17:25:29,298 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:29,299 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:29,299 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:37,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:37,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:37,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:39,309 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:39,311 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 17:25:39,326 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:39,328 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:39,328 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:47,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:47,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:47,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:49,338 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-15 17:25:49,338 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-15 17:25:49,355 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:49,358 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:49,358 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:57,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.26 % 2021-12-15 17:25:57,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:25:57,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:25:59,368 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.26 % 2021-12-15 17:25:59,368 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.26 2021-12-15 17:25:59,385 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:25:59,388 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:25:59,388 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:07,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.26 % 2021-12-15 17:26:07,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:07,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:09,398 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.26 % 2021-12-15 17:26:09,398 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.26 2021-12-15 17:26:09,415 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:09,417 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:09,417 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:17,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:17,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:17,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:19,430 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:19,430 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.45 2021-12-15 17:26:19,450 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:19,453 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:19,453 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:27,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:27,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:27,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:29,464 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:29,464 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.45 2021-12-15 17:26:29,483 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:29,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:29,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:37,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:37,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:37,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:39,496 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:39,496 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.45 2021-12-15 17:26:39,513 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:39,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:39,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:47,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:47,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:47,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:49,526 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:49,526 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.45 2021-12-15 17:26:49,542 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:49,543 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:49,545 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:57,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.45 % 2021-12-15 17:26:57,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:26:57,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:26:59,555 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.65 % 2021-12-15 17:26:59,555 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.65 2021-12-15 17:26:59,571 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:26:59,573 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:26:59,573 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:07,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.65 % 2021-12-15 17:27:07,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:07,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:09,584 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.65 % 2021-12-15 17:27:09,585 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.65 2021-12-15 17:27:09,602 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:09,605 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:09,605 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:17,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.65 % 2021-12-15 17:27:17,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:17,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:19,615 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.65 % 2021-12-15 17:27:19,617 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.65 2021-12-15 17:27:19,631 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:19,634 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:19,634 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:27,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:27,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:27,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:29,645 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:29,645 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 17:27:29,663 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:29,664 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:29,664 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:37,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:37,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:37,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:39,676 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:39,676 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 17:27:39,691 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:39,694 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:39,694 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:47,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:47,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:47,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:49,707 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:49,707 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 17:27:49,723 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:49,726 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:49,726 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:57,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:57,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:27:57,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:27:59,739 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 17:27:59,740 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 17:27:59,756 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:27:59,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:27:59,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:07,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:07,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:07,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:09,769 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:09,769 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-15 17:28:09,789 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:09,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:09,793 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:17,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:17,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:17,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:19,805 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:19,806 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-15 17:28:19,823 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:19,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:19,826 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:27,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:27,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:27,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:29,838 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-15 17:28:29,839 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-15 17:28:29,858 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:29,861 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:29,861 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:37,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:37,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:37,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:39,875 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:39,875 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.43 2021-12-15 17:28:39,894 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:39,897 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:39,897 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:47,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:47,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:47,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:49,910 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:49,910 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.43 2021-12-15 17:28:49,930 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:49,934 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:49,934 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:57,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:58,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:28:58,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:28:59,946 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:28:59,947 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.43 2021-12-15 17:28:59,964 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:28:59,967 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:28:59,967 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:08,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:29:08,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:08,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:09,979 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:29:09,979 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.43 2021-12-15 17:29:09,993 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:29:09,996 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:29:09,996 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:18,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:29:18,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:18,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:20,009 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:29:20,009 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.43 2021-12-15 17:29:20,029 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:29:20,032 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:29:20,032 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:28,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.43 % 2021-12-15 17:29:28,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:28,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:30,046 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.62 % 2021-12-15 17:29:30,046 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.62 2021-12-15 17:29:30,065 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:29:30,068 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:29:30,068 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:38,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 17:29:38,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:38,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:40,085 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 17:29:40,085 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 17:29:40,108 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:29:40,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:29:40,111 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:48,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 17:29:48,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:48,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:50,121 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 17:29:50,121 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 17:29:50,138 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:29:50,140 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:29:50,140 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:29:58,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 17:29:58,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:29:58,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:00,150 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:00,150 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:00,165 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:00,167 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:00,168 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:08,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:08,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:08,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:10,177 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:10,178 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:10,194 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:10,196 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:10,197 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:18,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:18,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:18,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:20,207 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:20,207 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:20,223 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:20,224 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:20,226 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:28,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:28,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:28,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:30,234 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:30,236 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:30,252 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:30,253 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:30,253 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:38,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:38,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:38,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:40,263 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:40,265 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:40,279 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:40,282 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:40,282 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:48,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:48,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:48,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:50,296 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.02 % 2021-12-15 17:30:50,296 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.02 2021-12-15 17:30:50,313 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:30:50,316 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:30:50,316 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:30:58,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 17:30:58,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:30:58,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:00,329 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 17:31:00,329 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.21 2021-12-15 17:31:00,346 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:00,349 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:00,351 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:08,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 17:31:08,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:08,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:10,361 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 17:31:10,361 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.21 2021-12-15 17:31:10,380 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:10,381 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:10,381 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:18,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.41 % 2021-12-15 17:31:18,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:18,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:20,391 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.41 % 2021-12-15 17:31:20,392 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.41 2021-12-15 17:31:20,408 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:20,410 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:20,411 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:28,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:28,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:28,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:30,421 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:30,421 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 17:31:30,437 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:30,440 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:30,440 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:38,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:38,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:38,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:40,469 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:40,469 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 17:31:40,486 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:40,489 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:40,489 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:48,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:48,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:48,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:50,497 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:50,499 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 17:31:50,513 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:31:50,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:31:50,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:31:58,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:31:58,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:31:58,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:00,526 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.60 % 2021-12-15 17:32:00,526 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.6 2021-12-15 17:32:00,545 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:00,548 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:00,548 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:08,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.80 % 2021-12-15 17:32:08,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:08,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:10,558 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.80 % 2021-12-15 17:32:10,558 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.8 2021-12-15 17:32:10,572 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:10,575 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:10,575 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:18,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.80 % 2021-12-15 17:32:18,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:18,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:20,588 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.80 % 2021-12-15 17:32:20,588 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.8 2021-12-15 17:32:20,607 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:20,609 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:20,609 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:28,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:28,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:28,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:30,622 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:30,622 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.99 2021-12-15 17:32:30,645 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:30,651 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:30,651 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:38,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:38,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:38,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:40,664 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:40,664 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.99 2021-12-15 17:32:40,681 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:40,684 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:40,684 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:48,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:48,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:48,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:50,694 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-15 17:32:50,696 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 8.99 2021-12-15 17:32:50,710 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:32:50,713 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:32:50,713 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:32:58,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.19 % 2021-12-15 17:32:58,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:32:58,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:00,726 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.19 % 2021-12-15 17:33:00,726 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.19 2021-12-15 17:33:00,742 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:00,744 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:00,744 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:08,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.19 % 2021-12-15 17:33:08,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:08,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:10,755 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.19 % 2021-12-15 17:33:10,755 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.19 2021-12-15 17:33:10,772 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:10,775 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:10,775 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:18,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.19 % 2021-12-15 17:33:18,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:18,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:20,785 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 17:33:20,785 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.38 2021-12-15 17:33:20,801 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:20,802 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:20,803 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:28,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:28,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:28,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:30,813 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:30,813 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 17:33:30,829 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:30,832 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:30,832 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:38,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:38,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:38,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:40,844 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:40,844 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 17:33:40,858 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:40,861 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:40,861 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:48,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:48,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:48,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:50,872 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.58 % 2021-12-15 17:33:50,872 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.58 2021-12-15 17:33:50,888 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:33:50,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:33:50,891 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:33:58,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:33:58,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:33:58,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:00,901 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:34:00,901 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.78 2021-12-15 17:34:00,917 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:00,920 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:00,920 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:08,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:34:08,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:08,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:10,931 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:34:10,931 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.78 2021-12-15 17:34:10,948 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:10,953 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:10,953 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:18,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:34:18,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:18,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:20,964 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.78 % 2021-12-15 17:34:20,964 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.78 2021-12-15 17:34:20,980 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:20,983 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:20,984 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:28,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-15 17:34:28,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:28,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:30,993 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-15 17:34:30,994 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-15 17:34:31,009 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:31,012 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:31,012 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:38,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-15 17:34:38,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:38,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:41,023 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.17 % 2021-12-15 17:34:41,023 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.17 2021-12-15 17:34:41,039 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:41,040 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:41,040 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:48,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.17 % 2021-12-15 17:34:48,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:48,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:51,053 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.17 % 2021-12-15 17:34:51,053 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.17 2021-12-15 17:34:51,071 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:34:51,073 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:34:51,073 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:34:58,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 17:34:58,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:34:58,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:01,085 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 17:35:01,085 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.36 2021-12-15 17:35:01,101 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:01,105 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:01,105 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:08,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-15 17:35:08,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:08,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:11,115 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 17:35:11,115 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.75 2021-12-15 17:35:11,131 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:11,132 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:11,134 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:18,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 17:35:18,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:18,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:21,144 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 17:35:21,144 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.75 2021-12-15 17:35:21,164 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:21,167 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:21,167 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:28,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 17:35:28,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:28,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:31,177 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 17:35:31,177 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.75 2021-12-15 17:35:31,193 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:31,196 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:31,196 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:38,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:35:38,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:38,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:41,206 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:35:41,206 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.95 2021-12-15 17:35:41,221 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:41,224 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:41,224 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:48,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:35:48,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:48,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:51,236 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:35:51,236 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.95 2021-12-15 17:35:51,262 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:35:51,265 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:35:51,265 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:35:58,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:35:58,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:35:58,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:01,276 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:36:01,276 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.95 2021-12-15 17:36:01,292 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:01,293 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:01,293 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:08,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:36:08,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:08,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:11,308 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.95 % 2021-12-15 17:36:11,308 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 10.95 2021-12-15 17:36:11,326 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:11,329 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:11,329 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:18,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 17:36:18,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:18,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:21,341 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 17:36:21,342 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.34 2021-12-15 17:36:21,355 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:21,358 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:21,358 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:28,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 17:36:28,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:28,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:31,368 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.34 % 2021-12-15 17:36:31,368 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.34 2021-12-15 17:36:31,385 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:31,388 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:31,388 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:38,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:36:38,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:38,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:41,398 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:36:41,398 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.73 2021-12-15 17:36:41,415 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:41,418 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:41,418 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:48,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:36:48,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:48,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:51,428 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:36:51,428 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.73 2021-12-15 17:36:51,447 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:36:51,448 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:36:51,450 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:36:58,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:36:58,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:36:58,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:01,464 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:37:01,466 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.73 2021-12-15 17:37:01,486 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:01,490 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:01,490 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:08,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:37:08,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:08,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:11,503 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.73 % 2021-12-15 17:37:11,503 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 11.73 2021-12-15 17:37:11,523 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:11,526 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:11,528 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:18,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.93 % 2021-12-15 17:37:18,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:18,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:21,538 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-15 17:37:21,538 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.12 2021-12-15 17:37:21,555 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:21,558 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:21,558 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:28,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-15 17:37:28,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:28,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:31,571 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-15 17:37:31,571 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.12 2021-12-15 17:37:31,588 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:31,591 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:31,591 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:38,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.32 % 2021-12-15 17:37:39,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:39,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:41,601 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.32 % 2021-12-15 17:37:41,601 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.32 2021-12-15 17:37:41,619 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:41,621 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:41,622 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:49,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:37:49,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:49,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:51,632 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:37:51,632 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.51 2021-12-15 17:37:51,650 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:37:51,653 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:37:51,653 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:37:59,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:37:59,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:37:59,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:01,663 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:38:01,663 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.51 2021-12-15 17:38:01,678 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:01,681 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:01,681 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:09,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:38:09,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:09,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:11,697 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.51 % 2021-12-15 17:38:11,697 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.51 2021-12-15 17:38:11,717 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:11,720 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:11,720 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:19,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:19,081 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:19,081 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:21,730 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:21,732 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.71 2021-12-15 17:38:21,752 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:21,755 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:21,756 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:29,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:29,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:29,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:31,766 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:31,767 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.71 2021-12-15 17:38:31,782 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:31,786 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:31,786 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:39,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:39,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:39,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:41,796 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:41,796 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.71 2021-12-15 17:38:41,812 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:41,815 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:41,815 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:49,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.71 % 2021-12-15 17:38:49,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:49,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:51,825 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.90 % 2021-12-15 17:38:51,825 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 12.9 2021-12-15 17:38:51,842 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:38:51,845 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:38:51,845 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:38:59,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.10 % 2021-12-15 17:38:59,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:38:59,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:01,857 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.10 % 2021-12-15 17:39:01,857 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.1 2021-12-15 17:39:01,874 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:01,875 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:01,875 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:09,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.10 % 2021-12-15 17:39:09,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:09,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:11,888 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.49 % 2021-12-15 17:39:11,888 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.49 2021-12-15 17:39:11,905 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:11,907 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:11,907 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:19,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.49 % 2021-12-15 17:39:19,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:19,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:21,918 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.49 % 2021-12-15 17:39:21,918 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.49 2021-12-15 17:39:21,936 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:21,937 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:21,937 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:29,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.49 % 2021-12-15 17:39:29,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:29,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:31,947 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:31,947 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-15 17:39:31,964 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:31,967 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:31,967 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:39,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:39,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:39,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:41,977 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:41,977 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-15 17:39:41,996 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:41,997 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:41,997 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:49,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:49,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:49,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:52,009 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:52,009 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-15 17:39:52,026 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:39:52,029 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:39:52,029 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:39:59,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-15 17:39:59,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:39:59,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:02,042 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:02,042 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.88 2021-12-15 17:40:02,058 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:02,059 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:02,059 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:09,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:09,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:09,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:12,071 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:12,071 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.88 2021-12-15 17:40:12,086 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:12,089 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:12,091 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:19,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:19,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:19,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:22,101 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:22,101 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.88 2021-12-15 17:40:22,118 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:22,119 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:22,119 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:29,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:29,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:29,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:32,131 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.88 % 2021-12-15 17:40:32,131 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 13.88 2021-12-15 17:40:32,151 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:32,155 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:32,155 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:39,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.08 % 2021-12-15 17:40:39,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:39,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:42,167 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 17:40:42,167 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 17:40:42,183 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:42,186 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:42,187 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:49,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 17:40:49,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:49,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:52,196 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 17:40:52,197 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 17:40:52,213 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:40:52,217 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:40:52,217 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:40:59,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 17:40:59,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:40:59,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:02,227 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.47 % 2021-12-15 17:41:02,227 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.47 2021-12-15 17:41:02,243 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:02,246 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:02,246 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:09,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.47 % 2021-12-15 17:41:09,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:09,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:12,256 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.47 % 2021-12-15 17:41:12,256 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.47 2021-12-15 17:41:12,273 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:12,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:12,276 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:19,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.47 % 2021-12-15 17:41:19,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:19,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:22,285 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.47 % 2021-12-15 17:41:22,285 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.47 2021-12-15 17:41:22,302 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:22,303 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:22,305 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:29,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:29,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:29,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:32,322 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:32,322 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 17:41:32,341 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:32,342 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:32,344 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:39,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:39,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:39,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:42,354 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:42,355 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 17:41:42,371 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:42,374 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:42,374 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:49,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:49,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:49,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:52,384 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 17:41:52,384 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 17:41:52,400 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:41:52,401 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:41:52,401 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:41:59,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:41:59,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:41:59,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:02,411 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:02,411 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.05 2021-12-15 17:42:02,426 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:02,427 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:02,427 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:09,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:09,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:09,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:12,437 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:12,437 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.05 2021-12-15 17:42:12,454 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:12,457 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:12,457 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:19,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:19,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:19,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:22,467 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:22,467 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.05 2021-12-15 17:42:22,483 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:22,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:22,486 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:29,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:29,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:29,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:32,499 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:32,499 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.05 2021-12-15 17:42:32,515 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:32,517 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:32,517 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:39,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:39,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:39,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:42,528 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.05 % 2021-12-15 17:42:42,528 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.05 2021-12-15 17:42:42,543 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:42,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:42,546 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:49,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.25 % 2021-12-15 17:42:49,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:49,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:52,558 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 17:42:52,558 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.44 2021-12-15 17:42:52,575 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:42:52,578 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:42:52,578 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:42:59,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 17:42:59,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:42:59,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:02,589 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:02,589 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.64 2021-12-15 17:43:02,611 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:02,614 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:02,614 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:09,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:09,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:09,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:12,624 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:12,624 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.64 2021-12-15 17:43:12,644 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:12,647 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:12,647 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:19,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:19,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:19,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:22,658 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:22,658 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.64 2021-12-15 17:43:22,677 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:22,680 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:22,680 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:29,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:29,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:29,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:32,690 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:32,690 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.64 2021-12-15 17:43:32,710 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:32,717 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:32,717 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:39,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:39,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:39,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:42,729 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.64 % 2021-12-15 17:43:42,729 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.64 2021-12-15 17:43:42,746 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:42,749 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:42,750 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:49,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.83 % 2021-12-15 17:43:49,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:49,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:52,763 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.83 % 2021-12-15 17:43:52,763 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 15.83 2021-12-15 17:43:52,779 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:43:52,782 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:43:52,782 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:43:59,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.03 % 2021-12-15 17:43:59,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:43:59,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:02,793 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.03 % 2021-12-15 17:44:02,793 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.03 2021-12-15 17:44:02,812 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:02,815 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:02,815 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:09,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.03 % 2021-12-15 17:44:09,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:09,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:12,825 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.03 % 2021-12-15 17:44:12,826 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.03 2021-12-15 17:44:12,845 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:12,848 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:12,848 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:19,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:19,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:19,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:22,858 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:22,859 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.23 2021-12-15 17:44:22,875 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:22,878 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:22,878 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:29,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:29,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:29,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:32,892 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:32,892 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.23 2021-12-15 17:44:32,911 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:32,914 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:32,914 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:39,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:39,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:39,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:42,924 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:42,924 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.23 2021-12-15 17:44:42,941 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:42,944 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:42,944 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:49,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.23 % 2021-12-15 17:44:49,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:49,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:52,957 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.42 % 2021-12-15 17:44:52,957 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.42 2021-12-15 17:44:52,974 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:44:52,977 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:44:52,977 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:44:59,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.62 % 2021-12-15 17:44:59,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:44:59,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:02,990 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 17:45:02,990 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 17:45:03,006 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:03,009 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:03,009 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:09,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 17:45:09,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:09,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:13,023 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 17:45:13,025 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 17:45:13,042 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:13,045 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:13,045 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:19,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.01 % 2021-12-15 17:45:19,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:19,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:23,055 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.01 % 2021-12-15 17:45:23,055 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.01 2021-12-15 17:45:23,071 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:23,073 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:23,073 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:29,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.01 % 2021-12-15 17:45:29,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:29,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:33,084 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.01 % 2021-12-15 17:45:33,084 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.01 2021-12-15 17:45:33,101 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:33,102 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:33,104 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:39,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.01 % 2021-12-15 17:45:39,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:39,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:43,114 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:45:43,115 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 17:45:43,131 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:43,134 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:43,134 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:49,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:45:49,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:49,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:53,144 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:45:53,145 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 17:45:53,161 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:45:53,164 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:45:53,164 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:45:59,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:45:59,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:45:59,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:03,177 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:46:03,177 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 17:46:03,196 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:03,197 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:03,198 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:09,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:46:10,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:46:10,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:13,207 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:46:13,209 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.2 2021-12-15 17:46:13,224 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:13,227 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:13,227 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:20,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.20 % 2021-12-15 17:46:20,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:46:20,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:23,239 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.40 % 2021-12-15 17:46:23,239 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.4 2021-12-15 17:46:23,255 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:23,256 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:23,256 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:30,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.40 % 2021-12-15 17:46:30,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:46:30,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:33,266 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.59 % 2021-12-15 17:46:33,266 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.59 2021-12-15 17:46:33,282 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:33,285 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:33,285 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:40,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.79 % 2021-12-15 17:46:40,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:46:40,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:43,296 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:46:43,296 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:46:43,312 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:43,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:43,315 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:50,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:46:50,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:46:50,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:46:53,325 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:46:53,325 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:46:53,342 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:46:53,344 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:46:53,345 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:00,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:00,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:00,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:03,357 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:03,357 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:47:03,375 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:03,377 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:03,377 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:10,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:10,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:10,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:13,385 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:13,387 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:47:13,403 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:13,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:13,404 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:20,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:20,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:20,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:23,414 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:23,414 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:47:23,430 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:23,431 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:23,431 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:30,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:30,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:30,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:33,441 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.99 % 2021-12-15 17:47:33,441 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 17.99 2021-12-15 17:47:33,457 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:33,459 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:33,460 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:40,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.38 % 2021-12-15 17:47:40,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:40,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:43,469 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.38 % 2021-12-15 17:47:43,470 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.38 2021-12-15 17:47:43,484 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:43,487 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:43,487 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:50,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.77 % 2021-12-15 17:47:50,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:47:50,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:47:53,497 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.77 % 2021-12-15 17:47:53,497 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.77 2021-12-15 17:47:53,515 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:47:53,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:47:53,516 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:00,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.77 % 2021-12-15 17:48:00,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:00,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:03,528 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.77 % 2021-12-15 17:48:03,528 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.77 2021-12-15 17:48:03,546 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:03,548 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:03,549 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:10,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:10,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:10,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:13,558 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:13,559 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.96 2021-12-15 17:48:13,575 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:13,578 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:13,578 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:20,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:20,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:20,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:23,592 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:23,592 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.96 2021-12-15 17:48:23,608 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:23,611 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:23,611 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:30,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:30,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:30,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:33,625 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:33,625 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.96 2021-12-15 17:48:33,647 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:33,651 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:33,651 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:40,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:40,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:40,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:43,667 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.96 % 2021-12-15 17:48:43,667 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 18.96 2021-12-15 17:48:43,686 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:43,688 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:43,688 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:50,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.16 % 2021-12-15 17:48:50,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:48:50,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:48:53,700 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.16 % 2021-12-15 17:48:53,700 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.16 2021-12-15 17:48:53,720 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:48:53,723 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:48:53,723 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:00,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.16 % 2021-12-15 17:49:00,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:00,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:03,733 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.35 % 2021-12-15 17:49:03,733 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.35 2021-12-15 17:49:03,755 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:03,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:03,759 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:10,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.35 % 2021-12-15 17:49:10,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:10,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:13,769 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.35 % 2021-12-15 17:49:13,770 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.35 2021-12-15 17:49:13,786 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:13,789 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:13,789 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:20,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.55 % 2021-12-15 17:49:20,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:20,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:23,801 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.55 % 2021-12-15 17:49:23,801 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.55 2021-12-15 17:49:23,819 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:23,822 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:23,822 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:30,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.55 % 2021-12-15 17:49:30,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:30,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:33,832 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 17:49:33,832 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.74 2021-12-15 17:49:33,848 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:33,849 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:33,849 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:40,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-15 17:49:40,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:40,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:43,861 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-15 17:49:43,861 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 19.94 2021-12-15 17:49:43,878 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:43,882 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:43,882 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:50,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-15 17:49:50,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:49:50,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:49:53,892 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-15 17:49:53,892 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-15 17:49:53,908 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:49:53,911 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:49:53,911 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:50:00,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-15 17:50:00,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:50:00,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:50:03,924 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-15 17:50:03,926 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-15 17:50:03,944 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:50:03,947 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:50:03,947 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:50:10,441 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.13% in 2765.65399981 seconds 2021-12-15 17:50:10,451 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 17:50:10,451 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 17:50:10,453 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 17:50:10,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:50:10,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 17:50:10,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:50:10,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 17:50:10,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 17:50:10,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 17:50:10,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 17:50:10,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 17:50:10,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 17:50:10,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 17:50:10,938 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 17:50:10,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 17:50:10,996 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['150292 memcached\n'], []) 2021-12-15 17:50:10,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 17:50:11,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 17:50:11,082 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['150292 memcached\n'], []) 2021-12-15 17:50:11,084 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 17:50:11,085 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 17:50:11,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:50:11,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 17:50:11,428 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:50:11,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 17:50:11,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 17:50:11,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 17:50:11,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 17:50:11,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 17:50:11,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 17:50:11,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 17:50:11,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:50:11,572 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 17:50:11,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:50:11,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:11,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:11,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:11,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:11,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:11,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:11,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:11,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:11,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:12,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:12,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:12,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:12,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:12,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:12,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:12,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:12,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:12,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:12,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:50:12,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:50:12,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:12,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:12,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 17:50:12,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 17:50:12,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:13,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:13,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 17:50:13,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 17:50:13,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:50:13,194 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 17:50:13,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:50:13,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:13,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:50:13,290 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 17:50:13,290 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 17:50:13,293 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 17:50:13,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:50:13,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 17:50:13,859 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:50:13,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 17:50:13,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 17:50:13,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 17:50:13,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 17:50:13,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 17:50:13,957 | test | DEBUG | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-15 17:50:13,959 | test | INFO | pool-3-thread-25 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-15 17:50:13,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 17:50:13,976 | infra | DEBUG | pool-3-thread-25 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:50:13,979 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:50:13,979 | infra | DEBUG | pool-3-thread-25 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:50:13,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 17:50:14,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:50:14,017 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 17:50:14,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:50:14,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:14,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:14,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:14,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:14,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:14,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:14,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:14,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:14,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:14,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:14,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:14,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:14,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:14,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:14,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:15,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:15,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:15,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:15,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:15,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:50:15,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:50:15,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:50:15,164 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 17:50:15,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:50:15,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:15,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:50:15,207 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 17:50:15,207 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 17:50:15,210 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 17:50:15,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:50:15,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 17:50:15,782 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:50:15,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 17:50:15,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 17:50:15,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 17:50:15,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 17:50:15,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 17:50:15,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 17:50:15,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 17:50:15,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:50:15,921 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 17:50:15,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:50:15,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:16,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:50:16,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:16,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:50:16,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:16,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:50:16,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:16,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:50:16,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:16,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:50:16,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:16,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:16,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:16,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:50:16,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:17,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:50:17,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:17,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:50:17,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:50:17,117 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 17:50:17,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:50:17,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:50:17,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:50:17,171 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 17:50:17,171 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 17:50:17,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 17:50:17,178 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 17:50:23,989 | test | ERROR | pool-3-thread-25 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'6709f7615c329a25810f27e023fcaeab', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e26195d6f40ec18887cdefb6ebc8bfb0', u'status': u'notRunning'} - rebalance failed 2021-12-15 17:50:24,023 | test | INFO | pool-3-thread-25 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 17:50:24,023 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423823L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.823Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.9324.141>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,479,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.25504.136>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 479}},\n infinity]}}}}}.\nRebalance Operation Id = 1581d1a6dd1bd412275c54e7eb2b155a'} 2021-12-15 17:50:24,023 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423819L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.819Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 16:56:24.629084 150525 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 17:50:24,023 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423791L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.791Z', u'text': u'Worker <0.9489.141> (for action {move,{479,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.68\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.9324.141>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 479,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.25504.136>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 479}},\n infinity]}}}}'} 2021-12-15 17:50:24,025 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423777L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.777Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 17:50:24,025 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639616555099L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:02:35.099Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 17:50:24,025 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639616554977L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:02:34.977Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 17:50:24,025 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639616554788L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:02:34.788Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 1581d1a6dd1bd412275c54e7eb2b155a"} 2021-12-15 17:50:24,026 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1639616424517L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:00:24.517Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.65\' in 234 seconds.'} 2021-12-15 17:50:24,026 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1639616384524L, u'shortText': u'message', u'serverTime': u'2021-12-15T16:59:44.524Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.66\' in 193 seconds.'} 2021-12-15 17:50:24,026 | test | ERROR | pool-3-thread-25 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639616346719L, u'shortText': u'message', u'serverTime': u'2021-12-15T16:59:06.719Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 156 seconds.'} 2021-12-15 17:50:24,026 | test | ERROR | pool-3-thread-25 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'6709f7615c329a25810f27e023fcaeab', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e26195d6f40ec18887cdefb6ebc8bfb0', u'status': u'notRunning'} - rebalance failed 2021-12-15 17:51:17,194 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 17:51:17,194 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 17:51:17,206 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:17,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:17,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:19,226 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:19,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:19,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:21,246 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:21,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:21,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:23,266 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:23,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:23,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:25,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:25,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:25,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:27,303 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:27,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:27,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:29,326 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:29,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:29,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:31,345 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:31,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:31,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:33,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:33,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:33,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:35,384 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:35,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:35,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:37,403 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:37,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:37,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:39,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:39,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:39,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:41,438 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:41,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:41,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:43,457 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:43,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:43,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:45,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:45,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:45,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:47,494 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:47,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:47,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:49,513 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:49,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:49,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:51,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:51,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:51,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:53,549 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:53,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:53,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:55,572 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:55,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:55,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:57,592 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:57,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:57,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:51:59,614 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:51:59,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:51:59,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:01,632 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:01,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:01,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:04,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:04,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:04,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:06,960 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:06,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:06,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:08,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:08,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:08,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:11,003 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:11,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:11,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:13,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:13,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:13,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:15,043 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:15,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:15,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:17,063 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:17,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:17,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:19,082 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:19,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:19,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:21,104 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:21,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:21,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:23,124 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:23,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:23,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:25,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:25,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:25,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:27,165 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:27,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:27,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:29,188 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:29,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:29,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:31,210 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:31,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:31,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:33,229 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:33,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:33,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:35,247 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:35,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:35,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:37,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:37,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:37,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:39,296 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:39,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:39,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:41,319 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:41,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:41,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:43,338 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:43,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:43,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:45,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:45,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:45,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:47,381 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:47,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:47,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:49,401 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:49,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:49,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:51,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:51,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:51,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:53,443 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:53,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:53,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:55,464 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:55,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:55,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:57,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:57,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:57,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:52:59,509 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:52:59,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:52:59,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:01,530 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:01,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:01,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:03,553 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:03,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:03,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:05,575 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:05,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:05,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:07,627 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:07,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:07,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:09,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:09,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:09,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:11,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:11,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:11,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:13,691 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:13,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:13,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:15,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:15,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:15,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:17,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:17,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:17,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:19,756 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:19,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:19,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:21,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:21,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:21,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:23,796 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:23,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:23,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:25,818 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:25,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:25,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:27,838 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:27,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:27,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:29,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:29,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:29,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:31,878 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:31,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:31,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:33,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:33,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:33,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:35,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:35,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:35,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:37,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:37,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:37,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:39,963 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:39,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:39,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:41,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:41,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:41,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:44,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:44,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:44,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:46,019 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:46,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:46,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:48,040 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:48,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:48,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:50,062 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:50,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:50,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:52,082 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:52,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:52,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:54,102 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:54,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:54,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:56,124 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:56,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:56,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:53:58,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:53:58,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:53:58,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:54:00,167 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 17:54:00,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 17:54:00,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 17:54:02,186 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 17:54:02,197 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 17:54:02,197 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 17:54:02,209 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 17:54:02,220 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 17:54:02,220 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 17:54:02,229 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 17:54:02,230 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 17:54:02,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:54:02,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 17:54:02,665 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:54:02,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 17:54:02,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 17:54:02,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 17:54:02,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 17:54:02,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 17:54:02,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 17:54:02,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 17:54:02,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:54:02,786 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 17:54:02,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:54:02,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:02,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:02,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:02,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:02,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:02,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:02,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:03,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:03,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:03,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:03,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:03,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:03,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:03,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:03,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:03,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:03,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:03,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:03,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:54:03,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:54:03,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 17:54:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 17:54:03,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 17:54:03,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 17:54:03,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:03,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:03,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 17:54:03,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 17:54:03,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:54:04,005 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 17:54:04,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:54:04,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:04,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:54:04,056 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 17:54:04,056 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 17:54:04,059 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 17:54:04,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:54:04,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 17:54:04,529 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:54:04,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 17:54:04,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 17:54:04,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 17:54:04,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 17:54:04,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 17:54:04,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 17:54:04,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 17:54:04,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:54:04,637 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 17:54:04,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:54:04,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:04,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:04,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:04,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:04,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:04,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:04,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:04,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:04,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:04,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:04,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:04,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:04,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:04,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:04,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:05,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:05,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:05,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:05,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:05,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:05,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:05,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:05,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:05,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:05,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:05,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:05,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:54:05,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 17:54:05,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:54:05,328 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 17:54:05,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:54:05,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:54:05,367 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 17:54:05,368 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 17:54:05,371 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 17:54:05,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 17:54:05,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 17:54:05,796 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 17:54:05,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 17:54:05,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 17:54:05,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 17:54:05,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 17:54:05,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 17:54:05,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 17:54:05,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 17:54:05,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 17:54:05,920 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 17:54:05,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 17:54:05,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:05,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:06,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 17:54:06,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:06,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 17:54:06,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:06,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 17:54:06,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:06,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 17:54:06,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:06,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 17:54:06,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:06,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 17:54:06,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 17:54:06,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:06,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 17:54:06,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 17:54:06,630 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 17:54:06,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 17:54:06,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 17:54:06,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 17:54:06,667 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 17:54:06,667 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 17:54:06,668 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639616554.72 2021-12-15 17:54:06,670 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-15 17:54:06,703 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639619646.67 2021-12-15 17:54:06,703 | infra | INFO | pool-3-thread-26 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639619646.67' started 2021-12-15 17:54:06,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 17:54:06,717 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 17:54:06,730 | test | DEBUG | pool-3-thread-26 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-15 17:54:06,730 | test | DEBUG | pool-3-thread-26 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-15 17:54:06,786 | test | DEBUG | pool-3-thread-26 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 17:54:06,786 | test | INFO | pool-3-thread-26 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 3.3528650364 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 1.75023980649 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 3.33314784931 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 7.60473208072 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 17:54:06,795 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:06,796 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:06,812 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:06,815 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:06,815 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:54:16,828 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:16,829 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:16,845 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:16,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:16,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:54:26,862 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:26,862 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:26,878 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:26,881 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:26,881 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:54:36,894 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:36,894 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:36,914 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:36,917 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:36,917 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:54:46,927 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:46,927 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:46,941 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:46,944 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:46,944 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:54:56,953 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 17:54:56,953 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 17:54:56,969 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:54:56,970 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:54:56,970 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:06,980 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.27 % 2021-12-15 17:55:06,980 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.27 2021-12-15 17:55:06,996 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:06,997 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:06,997 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:17,009 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.74 % 2021-12-15 17:55:17,010 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 0.74 2021-12-15 17:55:17,030 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:17,032 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:17,032 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:27,042 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.02 % 2021-12-15 17:55:27,042 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.02 2021-12-15 17:55:27,058 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:27,059 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:27,059 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:37,071 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.49 % 2021-12-15 17:55:37,071 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.49 2021-12-15 17:55:37,086 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:37,088 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:37,088 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:47,098 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.49 % 2021-12-15 17:55:47,098 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.49 2021-12-15 17:55:47,114 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:47,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:47,115 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:55:57,127 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.49 % 2021-12-15 17:55:57,128 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.49 2021-12-15 17:55:57,144 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:55:57,145 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:55:57,145 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:06,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:06,729 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 1.76 2021-12-15 17:56:06,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:06,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:06,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:07,154 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:07,155 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 17:56:07,171 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:07,171 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:07,173 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:16,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:16,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:16,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:17,187 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:17,187 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 17:56:17,210 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:17,211 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:17,213 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:26,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:26,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:26,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:27,224 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 17:56:27,224 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 17:56:27,243 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:27,244 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:27,244 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:36,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.04 % 2021-12-15 17:56:36,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:36,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:37,253 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.04 % 2021-12-15 17:56:37,255 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.04 2021-12-15 17:56:37,267 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:37,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:37,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:46,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:56:46,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:46,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:47,282 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:56:47,282 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.26 2021-12-15 17:56:47,299 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:47,302 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:47,302 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:56,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:56:56,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:56:56,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:56:57,312 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:56:57,312 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.26 2021-12-15 17:56:57,329 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:56:57,332 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:56:57,332 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:06,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:57:06,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:06,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:07,342 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.26 % 2021-12-15 17:57:07,342 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.26 2021-12-15 17:57:07,361 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:07,362 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:07,362 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:16,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.53 % 2021-12-15 17:57:16,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:16,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:17,375 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.53 % 2021-12-15 17:57:17,375 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.53 2021-12-15 17:57:17,390 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:17,391 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:17,391 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:26,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.53 % 2021-12-15 17:57:26,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:26,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:27,404 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.53 % 2021-12-15 17:57:27,404 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.53 2021-12-15 17:57:27,420 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:27,423 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:27,423 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:36,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.81 % 2021-12-15 17:57:36,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:36,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:37,437 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.81 % 2021-12-15 17:57:37,437 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 2.81 2021-12-15 17:57:37,453 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:37,456 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:37,456 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:46,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.81 % 2021-12-15 17:57:46,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:46,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:47,467 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.03 % 2021-12-15 17:57:47,467 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.03 2021-12-15 17:57:47,482 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:47,484 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:47,484 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:56,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.03 % 2021-12-15 17:57:57,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:57:57,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:57:57,494 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.03 % 2021-12-15 17:57:57,494 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.03 2021-12-15 17:57:57,513 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:57:57,516 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:57:57,516 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:07,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:07,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:07,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:07,526 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:07,526 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.3 2021-12-15 17:58:07,542 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:07,543 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:07,543 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:17,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:17,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:17,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:17,553 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:17,555 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.3 2021-12-15 17:58:17,572 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:17,575 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:17,575 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:27,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:27,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:27,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:27,589 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:27,589 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.3 2021-12-15 17:58:27,611 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:27,615 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:27,615 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:37,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:37,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:37,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:37,627 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.30 % 2021-12-15 17:58:37,628 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.3 2021-12-15 17:58:37,644 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:37,648 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:37,648 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:47,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:58:47,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:47,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:47,658 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 17:58:47,658 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 17:58:47,674 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:47,677 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:47,677 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:57,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.80 % 2021-12-15 17:58:57,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:58:57,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:58:57,690 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.80 % 2021-12-15 17:58:57,690 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 3.8 2021-12-15 17:58:57,706 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:58:57,709 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:58:57,709 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:07,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.27 % 2021-12-15 17:59:07,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:07,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:07,720 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.27 % 2021-12-15 17:59:07,720 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 4.27 2021-12-15 17:59:07,740 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:07,743 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:07,743 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:17,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:17,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:17,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:17,756 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:17,756 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 4.54 2021-12-15 17:59:17,773 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:17,776 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:17,776 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:27,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:27,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:27,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:27,789 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:27,789 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 4.54 2021-12-15 17:59:27,805 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:27,809 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:27,809 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:37,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:37,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:37,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:37,823 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.54 % 2021-12-15 17:59:37,823 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 4.54 2021-12-15 17:59:37,842 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:37,845 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:37,845 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:47,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.79 % 2021-12-15 17:59:47,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:47,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:47,857 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.79 % 2021-12-15 17:59:47,857 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 4.79 2021-12-15 17:59:47,872 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:47,875 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:47,875 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:57,193 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.01 % 2021-12-15 17:59:57,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 17:59:57,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 17:59:57,885 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.01 % 2021-12-15 17:59:57,887 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.01 2021-12-15 17:59:57,901 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 17:59:57,904 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 17:59:57,905 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:07,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:07,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:07,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:07,914 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:07,914 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.29 2021-12-15 18:00:07,930 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:07,934 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:07,934 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:17,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:17,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:17,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:17,944 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:17,944 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.29 2021-12-15 18:00:17,960 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:17,963 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:17,963 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:27,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:27,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:27,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:27,973 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:27,973 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.29 2021-12-15 18:00:27,989 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:27,990 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:27,990 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:37,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:37,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:37,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:38,002 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.29 % 2021-12-15 18:00:38,002 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.29 2021-12-15 18:00:38,022 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:38,025 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:38,025 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:47,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:00:47,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:47,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:48,035 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:00:48,035 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.81 2021-12-15 18:00:48,051 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:48,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:48,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:57,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:00:57,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:00:57,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:00:58,062 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:00:58,062 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.81 2021-12-15 18:00:58,075 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:00:58,078 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:00:58,078 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:07,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:01:07,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:07,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:08,088 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.81 % 2021-12-15 18:01:08,088 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 5.81 2021-12-15 18:01:08,104 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:08,105 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:08,105 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:17,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.03 % 2021-12-15 18:01:17,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:17,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:18,115 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.03 % 2021-12-15 18:01:18,115 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.03 2021-12-15 18:01:18,130 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:18,131 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:18,131 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:27,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.31 % 2021-12-15 18:01:27,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:27,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:28,141 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.31 % 2021-12-15 18:01:28,141 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.31 2021-12-15 18:01:28,157 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:28,160 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:28,160 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:37,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.31 % 2021-12-15 18:01:37,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:37,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:38,170 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.31 % 2021-12-15 18:01:38,171 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.31 2021-12-15 18:01:38,188 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:38,190 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:38,191 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:47,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.58 % 2021-12-15 18:01:47,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:47,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:48,201 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.58 % 2021-12-15 18:01:48,201 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.58 2021-12-15 18:01:48,217 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:48,219 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:48,219 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:57,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.80 % 2021-12-15 18:01:57,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:01:57,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:01:58,230 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.80 % 2021-12-15 18:01:58,232 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.8 2021-12-15 18:01:58,253 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:01:58,256 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:01:58,256 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:07,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.80 % 2021-12-15 18:02:07,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:07,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:08,269 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.80 % 2021-12-15 18:02:08,269 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 6.8 2021-12-15 18:02:08,286 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:08,288 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:08,289 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:17,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 18:02:17,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:17,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:18,301 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 18:02:18,301 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.02 2021-12-15 18:02:18,319 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:18,322 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:18,322 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:27,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.30 % 2021-12-15 18:02:27,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:27,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:28,332 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.30 % 2021-12-15 18:02:28,332 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.3 2021-12-15 18:02:28,348 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:28,349 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:28,349 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:37,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.30 % 2021-12-15 18:02:37,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:37,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:38,361 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.30 % 2021-12-15 18:02:38,361 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.3 2021-12-15 18:02:38,378 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:38,380 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:38,380 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:47,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.55 % 2021-12-15 18:02:47,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:47,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:48,390 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.55 % 2021-12-15 18:02:48,390 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.55 2021-12-15 18:02:48,405 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:48,407 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:48,407 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:57,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:02:57,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:02:57,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:02:58,417 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:02:58,417 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 18:02:58,431 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:02:58,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:02:58,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:07,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:07,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:07,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:08,444 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:08,446 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 18:03:08,464 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:08,467 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:08,467 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:17,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:17,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:17,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:18,480 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:18,482 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 18:03:18,500 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:18,503 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:18,503 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:27,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:27,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:27,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:28,515 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.82 % 2021-12-15 18:03:28,515 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 7.82 2021-12-15 18:03:28,533 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:28,538 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:28,538 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:37,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.04 % 2021-12-15 18:03:37,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:37,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:38,548 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.04 % 2021-12-15 18:03:38,549 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 8.04 2021-12-15 18:03:38,565 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:38,568 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:38,568 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:47,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.32 % 2021-12-15 18:03:47,630 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:47,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:48,578 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.32 % 2021-12-15 18:03:48,579 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 8.32 2021-12-15 18:03:48,596 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:48,599 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:48,599 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:57,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:03:57,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:03:57,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:03:58,611 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:03:58,611 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 8.84 2021-12-15 18:03:58,628 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:03:58,631 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:03:58,631 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:07,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:04:07,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:07,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:08,640 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:04:08,641 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 8.84 2021-12-15 18:04:08,657 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:08,660 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:08,660 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:17,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:04:17,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:17,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:18,681 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.84 % 2021-12-15 18:04:18,681 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 8.84 2021-12-15 18:04:18,698 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:18,701 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:18,701 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:27,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.10 % 2021-12-15 18:04:27,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:27,706 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:28,711 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.10 % 2021-12-15 18:04:28,711 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.1 2021-12-15 18:04:28,732 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:28,734 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:28,736 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:37,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.10 % 2021-12-15 18:04:37,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:37,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:38,747 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.10 % 2021-12-15 18:04:38,749 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.1 2021-12-15 18:04:38,766 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:38,767 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:38,767 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:47,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 18:04:47,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:47,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:48,779 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 18:04:48,779 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.38 2021-12-15 18:04:48,796 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:48,801 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:48,802 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:57,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.60 % 2021-12-15 18:04:57,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:04:57,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:04:58,812 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.60 % 2021-12-15 18:04:58,813 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.6 2021-12-15 18:04:58,829 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:04:58,832 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:04:58,832 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:07,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 18:05:07,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:07,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:08,844 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 18:05:08,845 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.87 2021-12-15 18:05:08,859 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:08,862 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:08,862 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:17,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 18:05:17,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:17,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:18,872 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 18:05:18,874 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 9.87 2021-12-15 18:05:18,888 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:18,891 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:18,891 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:27,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 18:05:27,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:27,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:28,901 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 18:05:28,901 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.08 2021-12-15 18:05:28,918 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:28,921 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:28,921 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:37,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 18:05:37,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:37,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:38,930 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-15 18:05:38,930 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.08 2021-12-15 18:05:38,946 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:38,948 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:38,948 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:47,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 18:05:47,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:47,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:48,959 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 18:05:48,959 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.36 2021-12-15 18:05:48,976 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:48,977 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:48,979 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:57,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 18:05:57,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:05:57,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:05:59,005 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.36 % 2021-12-15 18:05:59,006 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.36 2021-12-15 18:05:59,020 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:05:59,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:05:59,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:07,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:07,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:07,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:09,033 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:09,035 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.83 2021-12-15 18:06:09,051 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:09,052 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:09,053 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:17,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:17,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:17,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:19,065 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:19,065 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.83 2021-12-15 18:06:19,085 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:19,089 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:19,091 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:27,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:27,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:27,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:29,101 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.83 % 2021-12-15 18:06:29,101 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 10.83 2021-12-15 18:06:29,117 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:29,118 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:29,118 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:37,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.10 % 2021-12-15 18:06:37,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:37,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:39,140 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.10 % 2021-12-15 18:06:39,140 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.1 2021-12-15 18:06:39,161 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:39,163 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:39,163 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:47,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.10 % 2021-12-15 18:06:47,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:47,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:49,174 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.10 % 2021-12-15 18:06:49,174 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.1 2021-12-15 18:06:49,190 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:49,191 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:49,191 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:58,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.38 % 2021-12-15 18:06:58,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:06:58,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:06:59,201 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.38 % 2021-12-15 18:06:59,201 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.38 2021-12-15 18:06:59,217 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:06:59,220 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:06:59,220 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:08,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.60 % 2021-12-15 18:07:08,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:08,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:09,230 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.60 % 2021-12-15 18:07:09,230 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.6 2021-12-15 18:07:09,243 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:09,246 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:09,246 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:18,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.60 % 2021-12-15 18:07:18,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:18,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:19,257 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.60 % 2021-12-15 18:07:19,257 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.6 2021-12-15 18:07:19,273 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:19,275 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:19,276 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:28,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.86 % 2021-12-15 18:07:28,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:28,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:29,286 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.86 % 2021-12-15 18:07:29,286 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 11.86 2021-12-15 18:07:29,303 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:29,305 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:29,306 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:38,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.86 % 2021-12-15 18:07:38,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:38,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:39,318 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.13 % 2021-12-15 18:07:39,318 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.13 2021-12-15 18:07:39,339 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:39,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:39,342 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:48,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.13 % 2021-12-15 18:07:48,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:48,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:49,355 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.13 % 2021-12-15 18:07:49,357 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.13 2021-12-15 18:07:49,372 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:49,375 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:49,375 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:58,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.41 % 2021-12-15 18:07:58,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:07:58,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:07:59,385 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.41 % 2021-12-15 18:07:59,385 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.41 2021-12-15 18:07:59,401 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:07:59,403 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:07:59,403 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:08,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:08,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:08,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:09,413 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:09,414 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 18:08:09,433 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:09,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:09,434 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:18,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:18,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:18,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:19,444 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:19,444 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 18:08:19,463 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:19,464 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:19,464 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:28,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:28,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:28,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:29,476 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:29,477 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 18:08:29,493 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:29,496 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:29,496 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:38,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:38,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:38,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:39,506 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 18:08:39,506 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 18:08:39,523 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:39,526 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:39,526 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:48,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.90 % 2021-12-15 18:08:48,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:48,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:49,538 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.90 % 2021-12-15 18:08:49,538 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 12.9 2021-12-15 18:08:49,555 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:49,558 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:49,558 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:58,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.43 % 2021-12-15 18:08:58,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:08:58,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:08:59,569 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.43 % 2021-12-15 18:08:59,569 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 13.43 2021-12-15 18:08:59,585 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:08:59,588 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:08:59,588 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:08,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.64 % 2021-12-15 18:09:08,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:08,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:09,605 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.64 % 2021-12-15 18:09:09,605 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 13.64 2021-12-15 18:09:09,622 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:09,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:09,625 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:18,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.64 % 2021-12-15 18:09:18,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:18,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:19,637 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.64 % 2021-12-15 18:09:19,638 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 13.64 2021-12-15 18:09:19,653 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:19,655 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:19,655 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:28,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.86 % 2021-12-15 18:09:28,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:28,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:29,664 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.86 % 2021-12-15 18:09:29,665 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 13.86 2021-12-15 18:09:29,680 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:29,683 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:29,684 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:38,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-15 18:09:38,313 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:38,313 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:39,694 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-15 18:09:39,694 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.12 2021-12-15 18:09:39,710 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:39,713 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:39,713 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:48,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-15 18:09:48,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:48,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:49,721 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-15 18:09:49,723 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.12 2021-12-15 18:09:49,739 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:49,742 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:49,742 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:58,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.38 % 2021-12-15 18:09:58,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:09:58,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:09:59,750 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.38 % 2021-12-15 18:09:59,752 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.38 2021-12-15 18:09:59,786 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:09:59,789 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:09:59,789 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:08,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:08,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:08,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:09,801 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:09,801 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 18:10:09,819 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:09,822 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:09,822 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:18,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:18,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:18,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:19,841 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:19,841 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 18:10:19,855 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:19,858 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:19,858 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:28,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:28,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:28,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:29,868 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:29,868 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 18:10:29,884 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:29,888 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:29,888 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:38,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:38,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:38,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:39,900 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 18:10:39,900 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 18:10:39,917 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:39,918 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:39,918 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:48,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.89 % 2021-12-15 18:10:48,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:48,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:49,930 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.89 % 2021-12-15 18:10:49,930 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.89 2021-12-15 18:10:49,947 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:49,950 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:49,950 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:58,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.89 % 2021-12-15 18:10:58,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:10:58,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:10:59,960 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.89 % 2021-12-15 18:10:59,960 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 14.89 2021-12-15 18:10:59,976 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:10:59,980 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:10:59,980 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:08,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:08,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:08,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:09,990 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:09,990 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 18:11:10,006 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:11:10,009 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:11:10,009 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:18,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:18,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:18,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:20,019 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:20,019 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 18:11:20,036 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:11:20,039 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:11:20,039 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:28,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:28,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:28,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:30,049 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:30,049 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 18:11:30,075 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:11:30,078 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:11:30,078 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:38,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:38,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:38,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:40,088 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:40,089 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 18:11:40,105 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:11:40,108 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:11:40,108 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:48,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:48,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:48,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:50,118 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.68 % 2021-12-15 18:11:50,118 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.68 2021-12-15 18:11:50,135 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:11:50,138 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:11:50,138 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:11:58,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.91 % 2021-12-15 18:11:58,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:11:58,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:00,148 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.91 % 2021-12-15 18:12:00,148 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.91 2021-12-15 18:12:00,164 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:00,167 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:00,167 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:08,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.91 % 2021-12-15 18:12:08,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:08,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:10,177 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.91 % 2021-12-15 18:12:10,178 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 15.91 2021-12-15 18:12:10,196 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:10,198 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:10,198 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:18,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.45 % 2021-12-15 18:12:18,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:18,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:20,210 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.45 % 2021-12-15 18:12:20,211 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 16.45 2021-12-15 18:12:20,229 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:20,230 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:20,232 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:28,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.45 % 2021-12-15 18:12:28,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:28,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:30,244 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.45 % 2021-12-15 18:12:30,244 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 16.45 2021-12-15 18:12:30,259 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:30,263 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:30,263 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:38,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.45 % 2021-12-15 18:12:38,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:38,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:40,272 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-15 18:12:40,272 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-15 18:12:40,286 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:40,288 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:40,289 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:48,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-15 18:12:48,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:48,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:50,299 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-15 18:12:50,299 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-15 18:12:50,315 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:12:50,318 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:12:50,318 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:12:58,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-15 18:12:58,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:12:58,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:00,328 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-15 18:13:00,328 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-15 18:13:00,355 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:00,357 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:00,357 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:08,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.15 % 2021-12-15 18:13:08,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:08,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:10,367 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.15 % 2021-12-15 18:13:10,367 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.15 2021-12-15 18:13:10,387 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:10,388 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:10,388 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:18,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.42 % 2021-12-15 18:13:18,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:18,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:20,400 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:20,400 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 18:13:20,417 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:20,420 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:20,420 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:28,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:28,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:28,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:30,430 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:30,430 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 18:13:30,447 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:30,450 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:30,450 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:38,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:38,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:38,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:40,459 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:40,460 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 18:13:40,474 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:40,477 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:40,477 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:48,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:48,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:48,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:50,487 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:50,487 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.69 2021-12-15 18:13:50,503 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:13:50,506 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:13:50,506 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:13:58,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.69 % 2021-12-15 18:13:58,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:13:58,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:00,517 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 18:14:00,519 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.94 2021-12-15 18:14:00,536 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:00,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:00,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:08,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 18:14:08,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:08,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:10,549 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.94 % 2021-12-15 18:14:10,549 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 17.94 2021-12-15 18:14:10,568 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:10,569 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:10,571 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:18,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 18:14:18,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:18,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:20,582 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 18:14:20,582 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-15 18:14:20,599 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:20,602 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:20,602 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:28,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:28,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:28,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:30,612 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:30,612 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.71 2021-12-15 18:14:30,628 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:30,631 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:30,631 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:38,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:38,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:38,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:40,641 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:40,641 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.71 2021-12-15 18:14:40,661 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:40,664 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:40,664 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:48,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:48,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:48,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:50,674 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.71 % 2021-12-15 18:14:50,676 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.71 2021-12-15 18:14:50,691 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:14:50,694 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:14:50,694 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:14:58,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.97 % 2021-12-15 18:14:59,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:14:59,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:00,703 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.97 % 2021-12-15 18:15:00,704 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.97 2021-12-15 18:15:00,720 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:00,723 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:00,723 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:09,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.97 % 2021-12-15 18:15:09,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:09,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:10,734 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.97 % 2021-12-15 18:15:10,734 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 18.97 2021-12-15 18:15:10,750 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:10,752 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:10,753 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:19,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.25 % 2021-12-15 18:15:19,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:19,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:20,763 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.25 % 2021-12-15 18:15:20,763 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.25 2021-12-15 18:15:20,780 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:20,785 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:20,785 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:29,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.25 % 2021-12-15 18:15:29,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:29,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:30,798 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.25 % 2021-12-15 18:15:30,798 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.25 2021-12-15 18:15:30,816 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:30,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:30,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:39,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:15:39,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:39,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:40,829 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:15:40,829 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.74 2021-12-15 18:15:40,845 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:40,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:40,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:49,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:15:49,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:49,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:50,858 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:15:50,858 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.74 2021-12-15 18:15:50,875 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:15:50,877 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:15:50,877 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:15:59,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:15:59,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:15:59,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:00,888 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.74 % 2021-12-15 18:16:00,888 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.74 2021-12-15 18:16:00,904 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:00,907 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:00,907 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:09,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-15 18:16:09,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:09,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:10,917 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-15 18:16:10,917 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.99 2021-12-15 18:16:10,934 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:10,936 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:10,936 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:19,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-15 18:16:19,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:19,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:20,947 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-15 18:16:20,947 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 19.99 2021-12-15 18:16:20,963 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:20,966 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:20,967 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:29,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.20 % 2021-12-15 18:16:29,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:29,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:30,976 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.48 % 2021-12-15 18:16:30,977 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.48 2021-12-15 18:16:30,994 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:30,999 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:30,999 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:39,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.70 % 2021-12-15 18:16:39,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:39,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:41,009 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.70 % 2021-12-15 18:16:41,009 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.7 2021-12-15 18:16:41,025 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:41,026 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:41,028 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:49,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.70 % 2021-12-15 18:16:49,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:49,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:51,036 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.70 % 2021-12-15 18:16:51,038 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.7 2021-12-15 18:16:51,053 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:16:51,056 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:16:51,056 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:16:59,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:16:59,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:16:59,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:01,073 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:01,073 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-15 18:17:01,096 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:01,101 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:01,101 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:09,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:09,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:09,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:11,115 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:11,115 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-15 18:17:11,137 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:11,141 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:11,141 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:19,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:19,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:19,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:21,151 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:21,151 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-15 18:17:21,168 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:21,171 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:21,171 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:29,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:29,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:29,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:31,181 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-15 18:17:31,183 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-15 18:17:31,200 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:31,203 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:31,203 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:39,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.25 % 2021-12-15 18:17:39,313 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:39,313 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:41,213 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.25 % 2021-12-15 18:17:41,213 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 21.25 2021-12-15 18:17:41,227 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:41,230 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:41,230 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:49,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.51 % 2021-12-15 18:17:49,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:49,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:51,242 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.73 % 2021-12-15 18:17:51,242 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 21.73 2021-12-15 18:17:51,259 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:17:51,262 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:17:51,262 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:17:59,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:17:59,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:17:59,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:01,273 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:01,273 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.0 2021-12-15 18:18:01,290 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:01,292 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:01,293 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:09,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:09,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:09,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:11,302 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:11,303 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.0 2021-12-15 18:18:11,319 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:11,321 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:11,321 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:19,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:19,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:19,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:21,331 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:21,331 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.0 2021-12-15 18:18:21,345 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:21,346 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:21,346 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:29,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:29,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:29,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:31,357 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.00 % 2021-12-15 18:18:31,357 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.0 2021-12-15 18:18:31,371 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:31,372 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:31,372 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:39,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:18:39,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:39,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:41,382 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:18:41,384 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.28 2021-12-15 18:18:41,398 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:41,398 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:41,398 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:49,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:18:49,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:49,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:51,410 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:18:51,410 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.28 2021-12-15 18:18:51,427 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:18:51,428 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:18:51,428 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:18:59,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:18:59,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:18:59,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:01,438 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.28 % 2021-12-15 18:19:01,438 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.28 2021-12-15 18:19:01,453 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:01,456 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:01,456 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:09,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.55 % 2021-12-15 18:19:09,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:09,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:11,466 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.80 % 2021-12-15 18:19:11,466 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 22.8 2021-12-15 18:19:11,480 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:11,482 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:11,483 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:19,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.02 % 2021-12-15 18:19:19,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:19,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:21,493 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.02 % 2021-12-15 18:19:21,493 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.02 2021-12-15 18:19:21,510 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:21,513 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:21,513 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:29,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.02 % 2021-12-15 18:19:29,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:29,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:31,523 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.02 % 2021-12-15 18:19:31,525 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.02 2021-12-15 18:19:31,542 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:31,543 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:31,543 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:39,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.02 % 2021-12-15 18:19:39,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:39,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:41,555 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.30 % 2021-12-15 18:19:41,555 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.3 2021-12-15 18:19:41,572 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:41,575 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:41,575 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:49,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.30 % 2021-12-15 18:19:49,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:49,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:51,585 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.30 % 2021-12-15 18:19:51,585 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.3 2021-12-15 18:19:51,601 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:19:51,602 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:19:51,604 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:19:59,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.55 % 2021-12-15 18:19:59,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:19:59,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:01,615 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.55 % 2021-12-15 18:20:01,615 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.55 2021-12-15 18:20:01,632 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:01,635 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:01,635 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:09,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.77 % 2021-12-15 18:20:09,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:09,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:11,647 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.77 % 2021-12-15 18:20:11,647 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 23.77 2021-12-15 18:20:11,664 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:11,667 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:11,667 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:19,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:19,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:19,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:21,678 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:21,678 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.26 2021-12-15 18:20:21,697 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:21,700 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:21,700 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:29,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:29,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:29,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:31,713 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:31,713 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.26 2021-12-15 18:20:31,729 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:31,733 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:31,733 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:39,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:39,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:39,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:41,743 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.26 % 2021-12-15 18:20:41,744 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.26 2021-12-15 18:20:41,762 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:41,765 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:41,765 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:49,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:20:49,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:49,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:51,773 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:20:51,775 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.54 2021-12-15 18:20:51,789 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:20:51,792 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:20:51,792 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:20:59,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:20:59,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:20:59,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:01,802 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:21:01,802 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.54 2021-12-15 18:21:01,816 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:01,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:01,819 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:09,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:21:09,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:09,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:11,829 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:21:11,829 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.54 2021-12-15 18:21:11,845 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:11,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:11,848 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:19,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.54 % 2021-12-15 18:21:19,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:19,706 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:21,861 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.79 % 2021-12-15 18:21:21,861 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.79 2021-12-15 18:21:21,877 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:21,881 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:21,881 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:29,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.79 % 2021-12-15 18:21:29,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:29,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:31,891 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.79 % 2021-12-15 18:21:31,891 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 24.79 2021-12-15 18:21:31,907 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:31,910 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:31,910 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:39,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.06 % 2021-12-15 18:21:39,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:39,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:41,921 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.06 % 2021-12-15 18:21:41,921 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.06 2021-12-15 18:21:41,936 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:41,937 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:41,937 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:49,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:21:49,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:49,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:51,947 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:21:51,947 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-15 18:21:51,963 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:21:51,966 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:21:51,966 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:21:59,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:21:59,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:21:59,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:01,974 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:01,976 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-15 18:22:01,990 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:01,993 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:01,993 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:09,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:09,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:09,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:12,006 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:12,006 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-15 18:22:12,020 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:12,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:12,023 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:19,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:19,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:19,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:22,036 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:22,036 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-15 18:22:22,056 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:22,061 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:22,061 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:29,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:29,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:29,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:32,072 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:32,072 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-15 18:22:32,088 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:32,091 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:32,091 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:39,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-15 18:22:39,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:39,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:42,099 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.83 % 2021-12-15 18:22:42,101 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 25.83 2021-12-15 18:22:42,115 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:42,118 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:42,119 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:49,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.83 % 2021-12-15 18:22:49,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:49,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:52,128 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.33 % 2021-12-15 18:22:52,128 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.33 2021-12-15 18:22:52,144 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:22:52,147 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:22:52,148 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:22:59,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:22:59,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:22:59,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:02,157 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:02,157 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.58 2021-12-15 18:23:02,173 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:02,176 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:02,176 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:09,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:09,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:09,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:12,193 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:12,194 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.58 2021-12-15 18:23:12,211 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:12,213 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:12,214 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:19,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:19,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:19,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:22,223 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:22,224 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.58 2021-12-15 18:23:22,240 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:22,242 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:22,242 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:29,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:29,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:29,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:32,250 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:32,252 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.58 2021-12-15 18:23:32,266 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:32,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:32,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:39,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.58 % 2021-12-15 18:23:39,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:39,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:42,279 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.80 % 2021-12-15 18:23:42,279 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.8 2021-12-15 18:23:42,295 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:42,298 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:42,298 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:49,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.80 % 2021-12-15 18:23:49,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:49,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:52,309 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.80 % 2021-12-15 18:23:52,309 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 26.8 2021-12-15 18:23:52,325 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:23:52,326 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:23:52,326 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:23:59,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.06 % 2021-12-15 18:23:59,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:23:59,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:02,336 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.34 % 2021-12-15 18:24:02,336 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 27.34 2021-12-15 18:24:02,351 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:02,354 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:02,354 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:10,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.34 % 2021-12-15 18:24:10,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:24:10,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:12,365 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.34 % 2021-12-15 18:24:12,365 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 27.34 2021-12-15 18:24:12,384 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:12,385 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:12,385 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:20,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.57 % 2021-12-15 18:24:20,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:24:20,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:22,395 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.57 % 2021-12-15 18:24:22,397 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 27.57 2021-12-15 18:24:22,411 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:22,413 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:22,413 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:30,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:30,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:24:30,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:32,424 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:32,424 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.07 2021-12-15 18:24:32,447 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:32,450 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:32,450 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:40,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:40,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:24:40,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:42,460 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:42,460 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.07 2021-12-15 18:24:42,476 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:42,479 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:42,479 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:50,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:50,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:24:50,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:24:52,489 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:24:52,490 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.07 2021-12-15 18:24:52,506 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:24:52,509 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:24:52,509 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:00,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.07 % 2021-12-15 18:25:00,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:00,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:02,519 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.34 % 2021-12-15 18:25:02,520 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.34 2021-12-15 18:25:02,536 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:02,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:02,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:10,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.62 % 2021-12-15 18:25:10,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:10,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:12,552 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.62 % 2021-12-15 18:25:12,552 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.62 2021-12-15 18:25:12,568 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:12,571 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:12,571 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:20,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.83 % 2021-12-15 18:25:20,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:20,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:22,582 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.83 % 2021-12-15 18:25:22,582 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 28.83 2021-12-15 18:25:22,598 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:22,601 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:22,601 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:30,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:30,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:30,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:32,630 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:32,631 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 29.05 2021-12-15 18:25:32,650 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:32,651 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:32,651 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:40,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:40,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:40,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:42,665 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:42,665 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 29.05 2021-12-15 18:25:42,681 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:42,684 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:42,684 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:50,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:50,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:25:50,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:25:52,696 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:25:52,697 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 29.05 2021-12-15 18:25:52,713 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:25:52,716 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:25:52,716 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:00,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:26:00,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:00,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:02,726 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.05 % 2021-12-15 18:26:02,726 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 29.05 2021-12-15 18:26:02,742 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:02,743 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:02,743 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:10,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.11 % 2021-12-15 18:26:10,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:10,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:12,753 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.11 % 2021-12-15 18:26:12,753 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.11 2021-12-15 18:26:12,769 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:12,772 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:12,772 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:20,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.11 % 2021-12-15 18:26:20,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:20,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:22,786 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.11 % 2021-12-15 18:26:22,786 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.11 2021-12-15 18:26:22,802 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:22,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:22,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:30,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.11 % 2021-12-15 18:26:30,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:30,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:32,816 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.34 % 2021-12-15 18:26:32,816 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.34 2021-12-15 18:26:32,832 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:32,835 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:32,835 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:40,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:26:40,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:40,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:42,845 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:26:42,845 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.55 2021-12-15 18:26:42,861 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:42,865 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:42,865 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:50,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:26:50,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:26:50,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:26:52,875 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:26:52,877 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.55 2021-12-15 18:26:52,891 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:26:52,894 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:26:52,894 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:00,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:27:00,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:00,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:02,905 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.55 % 2021-12-15 18:27:02,907 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.55 2021-12-15 18:27:02,921 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:02,924 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:02,924 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:10,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.83 % 2021-12-15 18:27:10,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:10,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:12,937 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.83 % 2021-12-15 18:27:12,937 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 30.83 2021-12-15 18:27:12,954 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:12,959 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:12,959 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:20,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.10 % 2021-12-15 18:27:20,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:20,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:22,969 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.10 % 2021-12-15 18:27:22,969 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 31.1 2021-12-15 18:27:22,984 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:22,987 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:22,987 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:30,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.35 % 2021-12-15 18:27:30,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:30,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:32,996 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.35 % 2021-12-15 18:27:32,996 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 31.35 2021-12-15 18:27:33,013 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:33,015 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:33,016 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:40,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.35 % 2021-12-15 18:27:40,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:40,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:43,025 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.35 % 2021-12-15 18:27:43,026 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 31.35 2021-12-15 18:27:43,040 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:43,043 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:43,043 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:50,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.56 % 2021-12-15 18:27:50,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:27:50,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:27:53,053 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.56 % 2021-12-15 18:27:53,053 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 31.56 2021-12-15 18:27:53,071 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:27:53,073 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:27:53,073 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:00,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.04 % 2021-12-15 18:28:00,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:00,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:03,082 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.04 % 2021-12-15 18:28:03,082 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.04 2021-12-15 18:28:03,098 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:03,101 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:03,101 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:10,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.04 % 2021-12-15 18:28:10,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:10,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:13,111 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.32 % 2021-12-15 18:28:13,111 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.32 2021-12-15 18:28:13,131 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:13,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:13,134 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:20,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.32 % 2021-12-15 18:28:20,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:20,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:23,148 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.32 % 2021-12-15 18:28:23,148 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.32 2021-12-15 18:28:23,167 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:23,170 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:23,170 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:30,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.58 % 2021-12-15 18:28:30,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:30,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:33,180 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.58 % 2021-12-15 18:28:33,180 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.58 2021-12-15 18:28:33,198 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:33,203 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:33,203 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:40,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.58 % 2021-12-15 18:28:40,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:40,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:43,216 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.58 % 2021-12-15 18:28:43,217 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.58 2021-12-15 18:28:43,247 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:43,252 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:43,252 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:50,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.58 % 2021-12-15 18:28:50,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:28:50,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:28:53,266 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.84 % 2021-12-15 18:28:53,266 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.84 2021-12-15 18:28:53,288 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:28:53,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:28:53,290 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:00,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.84 % 2021-12-15 18:29:00,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:00,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:03,301 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.84 % 2021-12-15 18:29:03,302 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.84 2021-12-15 18:29:03,318 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:03,319 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:03,319 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:10,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.84 % 2021-12-15 18:29:10,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:10,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:13,329 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.84 % 2021-12-15 18:29:13,329 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 32.84 2021-12-15 18:29:13,344 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:13,346 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:13,346 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:20,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.05 % 2021-12-15 18:29:20,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:20,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:23,357 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.05 % 2021-12-15 18:29:23,357 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 33.05 2021-12-15 18:29:23,372 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:23,375 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:23,375 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:30,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.53 % 2021-12-15 18:29:30,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:30,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:33,385 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.80 % 2021-12-15 18:29:33,385 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 33.8 2021-12-15 18:29:33,403 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:33,405 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:33,405 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:40,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:29:40,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:40,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:43,417 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:29:43,417 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 18:29:43,434 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:43,436 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:43,437 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:50,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:29:50,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:29:50,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:29:53,447 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:29:53,447 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 18:29:53,463 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:29:53,466 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:29:53,466 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:00,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:00,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:00,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:03,476 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:03,476 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 18:30:03,493 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:03,510 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:03,510 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:10,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:10,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:10,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:13,520 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:13,520 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 18:30:13,536 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:13,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:13,539 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:20,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:20,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:20,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:23,551 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.08 % 2021-12-15 18:30:23,551 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.08 2021-12-15 18:30:23,566 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:23,569 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:23,569 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:30,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.35 % 2021-12-15 18:30:30,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:30,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:33,579 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.35 % 2021-12-15 18:30:33,579 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 34.35 2021-12-15 18:30:33,595 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:33,598 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:33,598 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:40,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.88 % 2021-12-15 18:30:40,767 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:40,767 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:43,608 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.09 % 2021-12-15 18:30:43,608 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.09 2021-12-15 18:30:43,625 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:43,627 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:43,627 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:50,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-15 18:30:50,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:30:50,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:30:53,638 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:30:53,638 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.57 2021-12-15 18:30:53,654 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:30:53,657 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:30:53,657 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:00,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:00,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:00,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:03,668 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:03,670 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.57 2021-12-15 18:31:03,687 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:03,690 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:03,690 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:10,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:10,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:10,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:13,700 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:13,700 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.57 2021-12-15 18:31:13,723 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:13,726 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:13,726 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:20,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:20,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:20,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:23,737 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:23,737 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.57 2021-12-15 18:31:23,756 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:23,760 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:23,760 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:30,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.57 % 2021-12-15 18:31:30,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:30,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:33,778 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:31:33,778 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 18:31:33,802 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:33,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:33,805 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:40,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:31:40,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:40,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:43,818 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:31:43,818 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 18:31:43,835 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:43,836 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:43,838 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:50,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:31:50,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:31:50,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:31:53,862 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:31:53,862 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 18:31:53,882 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:31:53,884 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:31:53,884 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:00,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:32:00,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:00,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:03,894 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:32:03,894 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 18:32:03,910 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:03,911 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:03,913 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:10,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:32:10,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:10,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:13,923 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.84 % 2021-12-15 18:32:13,923 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 35.84 2021-12-15 18:32:13,940 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:13,943 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:13,943 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:20,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.37 % 2021-12-15 18:32:20,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:20,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:23,953 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.37 % 2021-12-15 18:32:23,953 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.37 2021-12-15 18:32:23,969 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:23,971 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:23,971 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:30,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.37 % 2021-12-15 18:32:30,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:30,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:33,982 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.64 % 2021-12-15 18:32:33,982 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.64 2021-12-15 18:32:33,996 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:34,000 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:34,000 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:40,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.64 % 2021-12-15 18:32:41,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:41,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:44,010 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.64 % 2021-12-15 18:32:44,010 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.64 2021-12-15 18:32:44,026 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:44,029 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:44,029 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:51,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.64 % 2021-12-15 18:32:51,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:32:51,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:32:54,039 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:32:54,039 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.91 2021-12-15 18:32:54,055 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:32:54,056 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:32:54,056 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:01,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:01,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:01,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:04,066 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:04,066 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.91 2021-12-15 18:33:04,085 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:04,086 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:04,088 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:11,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:11,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:11,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:14,098 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:14,098 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.91 2021-12-15 18:33:14,117 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:14,118 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:14,118 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:21,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:21,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:21,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:24,128 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.91 % 2021-12-15 18:33:24,128 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 36.91 2021-12-15 18:33:24,145 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:24,148 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:24,148 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:31,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.45 % 2021-12-15 18:33:31,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:31,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:34,160 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.45 % 2021-12-15 18:33:34,160 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.45 2021-12-15 18:33:34,177 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:34,178 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:34,178 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:41,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.45 % 2021-12-15 18:33:41,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:41,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:44,190 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.45 % 2021-12-15 18:33:44,190 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.45 2021-12-15 18:33:44,206 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:44,209 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:44,210 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:51,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.66 % 2021-12-15 18:33:51,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:33:51,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:33:54,220 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.66 % 2021-12-15 18:33:54,220 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.66 2021-12-15 18:33:54,236 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:33:54,240 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:33:54,240 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:01,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.66 % 2021-12-15 18:34:01,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:01,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:04,250 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:04,250 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.93 2021-12-15 18:34:04,265 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:04,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:04,269 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:11,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:11,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:11,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:14,279 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:14,279 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.93 2021-12-15 18:34:14,296 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:14,299 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:14,299 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:21,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:21,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:21,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:24,309 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:24,309 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.93 2021-12-15 18:34:24,325 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:24,328 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:24,328 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:31,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:31,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:31,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:34,338 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.93 % 2021-12-15 18:34:34,338 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 37.93 2021-12-15 18:34:34,352 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:34,355 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:34,355 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:41,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-15 18:34:41,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:41,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:44,367 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-15 18:34:44,367 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.15 2021-12-15 18:34:44,381 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:44,384 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:44,384 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:51,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-15 18:34:51,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:34:51,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:34:54,394 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-15 18:34:54,394 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.15 2021-12-15 18:34:54,411 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:34:54,413 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:34:54,414 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:01,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.43 % 2021-12-15 18:35:01,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:01,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:04,424 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.43 % 2021-12-15 18:35:04,424 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.43 2021-12-15 18:35:04,441 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:04,444 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:04,444 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:11,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-15 18:35:11,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:11,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:14,453 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-15 18:35:14,453 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.7 2021-12-15 18:35:14,467 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:14,470 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:14,470 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:21,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-15 18:35:21,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:21,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:24,480 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-15 18:35:24,480 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.7 2021-12-15 18:35:24,494 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:24,496 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:24,496 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:31,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:31,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:31,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:34,506 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:34,506 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.95 2021-12-15 18:35:34,522 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:34,523 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:34,523 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:41,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:41,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:41,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:44,533 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:44,533 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.95 2021-12-15 18:35:44,549 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:44,552 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:44,552 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:51,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:51,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:35:51,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:35:54,562 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.95 % 2021-12-15 18:35:54,562 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 38.95 2021-12-15 18:35:54,578 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:35:54,579 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:35:54,579 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:01,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.45 % 2021-12-15 18:36:01,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:01,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:04,588 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.45 % 2021-12-15 18:36:04,588 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.45 2021-12-15 18:36:04,605 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:04,608 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:04,608 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:11,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.45 % 2021-12-15 18:36:11,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:11,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:14,618 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.45 % 2021-12-15 18:36:14,618 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.45 2021-12-15 18:36:14,634 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:14,635 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:14,635 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:21,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.72 % 2021-12-15 18:36:21,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:21,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:24,645 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.72 % 2021-12-15 18:36:24,645 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.72 2021-12-15 18:36:24,663 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:24,665 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:24,665 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:31,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.72 % 2021-12-15 18:36:31,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:31,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:34,676 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.72 % 2021-12-15 18:36:34,677 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.72 2021-12-15 18:36:34,690 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:34,693 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:34,693 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:41,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.72 % 2021-12-15 18:36:41,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:41,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:44,703 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.93 % 2021-12-15 18:36:44,703 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.93 2021-12-15 18:36:44,724 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:44,726 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:44,726 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:51,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.93 % 2021-12-15 18:36:51,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:36:51,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:36:54,740 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.93 % 2021-12-15 18:36:54,740 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 39.93 2021-12-15 18:36:54,753 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:36:54,756 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:36:54,756 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:37:01,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.16 % 2021-12-15 18:37:01,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:37:01,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:37:04,766 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.16 % 2021-12-15 18:37:04,766 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 40.16 2021-12-15 18:37:04,783 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:37:04,786 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:37:04,786 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:37:11,561 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.16% in 2464.83099985 seconds 2021-12-15 18:37:11,572 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 18:37:11,572 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 18:37:11,575 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 18:37:11,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:37:12,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 18:37:12,223 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:37:12,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 18:37:12,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 18:37:12,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 18:37:12,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 18:37:12,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 18:37:12,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 18:37:12,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 18:37:12,334 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 18:37:12,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 18:37:12,394 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['154818 memcached\n'], []) 2021-12-15 18:37:12,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 18:37:12,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 18:37:12,484 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['154818 memcached\n'], []) 2021-12-15 18:37:12,486 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 18:37:12,489 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 18:37:12,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:37:12,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 18:37:12,867 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:37:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 18:37:12,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 18:37:12,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 18:37:12,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 18:37:12,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 18:37:12,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 18:37:13,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 18:37:13,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:37:13,046 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 18:37:13,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:37:13,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:13,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:13,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:13,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:13,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:13,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:13,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:13,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:13,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:13,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:13,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:13,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:13,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:13,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:13,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:13,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:13,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:37:14,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:37:14,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:14,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:14,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 18:37:14,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 18:37:14,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:14,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:14,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 18:37:14,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 18:37:14,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:14,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:14,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 18:37:14,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 18:37:14,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:37:14,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 18:37:14,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:37:14,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:14,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:37:14,421 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 18:37:14,421 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 18:37:14,424 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 18:37:14,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:37:14,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 18:37:14,798 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.16 % 2021-12-15 18:37:14,798 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 40.16 2021-12-15 18:37:14,822 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:37:14,825 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:37:14,825 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:37:14,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:37:14,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 18:37:14,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 18:37:14,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 18:37:14,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 18:37:14,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 18:37:14,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 18:37:14,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 18:37:14,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:37:14,947 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 18:37:14,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:37:14,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:15,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:15,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:15,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:15,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:15,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:15,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:15,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:15,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:15,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:15,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:15,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:15,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:15,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:15,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:15,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:37:15,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:37:15,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:37:15,647 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 18:37:15,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:37:15,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:15,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:37:15,687 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 18:37:15,687 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 18:37:15,690 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 18:37:15,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:37:16,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 18:37:16,311 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:37:16,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 18:37:16,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 18:37:16,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 18:37:16,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 18:37:16,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 18:37:16,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 18:37:16,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 18:37:16,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:37:16,437 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 18:37:16,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:37:16,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:16,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:16,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:37:16,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:16,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:16,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:16,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:37:16,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:16,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:16,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:16,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:37:16,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:16,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:16,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:17,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:37:17,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:17,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:17,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:17,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:37:17,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:17,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:17,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:17,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:37:17,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:17,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:37:17,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:17,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:37:17,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:37:17,332 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 18:37:17,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:37:17,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:37:17,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:37:17,371 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 18:37:17,371 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 18:37:17,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 18:37:17,378 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 18:37:24,836 | test | DEBUG | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.16 % 2021-12-15 18:37:24,836 | test | INFO | pool-3-thread-26 | [task:check:385] Rebalance - status: running, progress: 40.16 2021-12-15 18:37:24,861 | infra | DEBUG | pool-3-thread-26 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:37:24,862 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:37:24,864 | infra | DEBUG | pool-3-thread-26 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:37:34,874 | test | ERROR | pool-3-thread-26 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fc9371618c01f192ee463e68baf95d41', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4e1a7ecbb70bc89597a300824f15cb11', u'status': u'notRunning'} - rebalance failed 2021-12-15 18:37:34,913 | test | INFO | pool-3-thread-26 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 18:37:34,913 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246114L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.114Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 17:50:32.409045 155048 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246110L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.110Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.31947.145>,\n {socket_closed,\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,<0.23175.141>,\n {update_vbucket_state,441,active,\n undefined,undefined,\n [['ns_1@172.23.110.64',\n 'ns_1@172.23.110.68'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66']]}},\n infinity]}}}}}.\nRebalance Operation Id = cfedf5aeb5e515813e5c79c2e1ff50f9"} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246094L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.094Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246067L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.067Z', u'text': u"Worker <0.27375.145> (for action {move,\n {441,\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.68'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66'],\n []}}) exited with reason {unexpected_exit,\n {'EXIT',\n <0.31947.145>,\n {socket_closed,\n {gen_server,\n call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,\n <0.23175.141>,\n {update_vbucket_state,\n 441,\n active,\n undefined,\n undefined,\n [['ns_1@172.23.110.64',\n 'ns_1@172.23.110.68'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66']]}},\n infinity]}}}}"} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619647150L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:54:07.150Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619646951L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:54:06.951Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 18:37:34,914 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619646734L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:54:06.734Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = cfedf5aeb5e515813e5c79c2e1ff50f9"} 2021-12-15 18:37:34,915 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619621127L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:53:41.127Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 158 seconds.'} 2021-12-15 18:37:34,915 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423823L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.823Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.9324.141>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,479,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.25504.136>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 479}},\n infinity]}}}}}.\nRebalance Operation Id = 1581d1a6dd1bd412275c54e7eb2b155a'} 2021-12-15 18:37:34,915 | test | ERROR | pool-3-thread-26 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639619423819L, u'shortText': u'message', u'serverTime': u'2021-12-15T17:50:23.819Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 16:56:24.629084 150525 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 18:37:34,917 | test | ERROR | pool-3-thread-26 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fc9371618c01f192ee463e68baf95d41', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4e1a7ecbb70bc89597a300824f15cb11', u'status': u'notRunning'} - rebalance failed 2021-12-15 18:38:17,391 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 18:38:17,391 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 18:38:17,403 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:17,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:17,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:19,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:19,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:19,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:21,440 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:21,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:21,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:23,460 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:23,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:23,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:25,480 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:25,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:25,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:27,500 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:27,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:27,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:29,520 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:29,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:29,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:31,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:31,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:31,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:33,565 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:33,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:33,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:35,586 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:35,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:35,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:37,608 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:37,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:37,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:39,630 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:39,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:39,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:41,651 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:41,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:41,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:43,671 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:43,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:43,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:45,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:45,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:45,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:47,713 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:47,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:47,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:49,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:49,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:49,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:51,753 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:51,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:51,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:53,776 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:53,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:53,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:55,801 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:55,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:55,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:57,828 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:57,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:57,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:38:59,848 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:38:59,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:38:59,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:01,868 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:01,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:01,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:03,888 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:03,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:03,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:05,914 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:05,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:05,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:07,934 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:07,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:07,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:09,953 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:09,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:09,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:11,973 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:11,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:11,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:13,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:13,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:13,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:16,007 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:16,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:16,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:18,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:18,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:18,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:20,042 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:20,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:20,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:22,062 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:22,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:22,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:24,082 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:24,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:24,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:28,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:28,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:28,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:30,507 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:30,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:30,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:32,532 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:32,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:32,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:34,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:34,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:34,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:36,576 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:36,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:36,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:38,598 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:38,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:38,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:40,618 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:40,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:40,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:42,637 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:42,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:42,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:44,661 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:44,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:44,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:46,684 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:46,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:46,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:48,707 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:48,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:48,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:50,727 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:50,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:50,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:52,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:52,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:52,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:54,767 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:54,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:54,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:56,792 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:56,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:56,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:39:58,816 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:39:58,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:39:58,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:00,838 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:00,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:00,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:02,861 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:02,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:02,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:04,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:04,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:04,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:06,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:06,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:06,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:08,956 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:08,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:08,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:10,977 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:10,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:10,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:12,999 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:13,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:13,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:15,019 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:15,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:15,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:17,039 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:17,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:17,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:19,058 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:19,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:19,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:21,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:21,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:21,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:23,095 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:23,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:23,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:25,115 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:25,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:25,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:27,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:27,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:27,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:29,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:29,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:29,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:31,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:31,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:31,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:33,200 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:33,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:33,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:35,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:35,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:35,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:37,242 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:37,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:37,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:39,260 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:39,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:39,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:41,280 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:41,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:41,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:43,319 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:43,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:43,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:45,339 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:45,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:45,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:47,359 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:47,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:47,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:49,380 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:49,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:49,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:51,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:51,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:51,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:53,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:53,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:53,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:55,443 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:55,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:55,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:57,463 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:57,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:57,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:40:59,484 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:40:59,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:40:59,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:01,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:01,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:01,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:03,523 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:03,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:03,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:05,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:05,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:05,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:07,563 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:07,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:07,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:09,585 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:09,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:09,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:11,605 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:11,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:11,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:13,625 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:13,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:13,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:15,647 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 18:41:15,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 18:41:15,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 18:41:17,664 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 18:41:17,677 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 18:41:17,678 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 18:41:17,688 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 18:41:17,700 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 18:41:17,700 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 18:41:17,709 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 18:41:17,710 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 18:41:17,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:41:18,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 18:41:18,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:41:18,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 18:41:18,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 18:41:18,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 18:41:18,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 18:41:18,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 18:41:18,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 18:41:18,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 18:41:18,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:41:18,290 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 18:41:18,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:41:18,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:18,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:18,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:18,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:18,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:18,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:18,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:18,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:18,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:18,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:18,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:18,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:18,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:18,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:18,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:18,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:18,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:18,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:18,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:18,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:18,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:19,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:19,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:19,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:19,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:41:19,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:41:19,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 18:41:19,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 18:41:19,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 18:41:19,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 18:41:19,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:19,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 18:41:19,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 18:41:19,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:41:19,530 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 18:41:19,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:41:19,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:19,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:41:19,575 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 18:41:19,575 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 18:41:19,578 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 18:41:19,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:41:19,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 18:41:19,938 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:41:19,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 18:41:19,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 18:41:19,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 18:41:19,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 18:41:20,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 18:41:20,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 18:41:20,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 18:41:20,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:41:20,068 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 18:41:20,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:41:20,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:20,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:20,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:20,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:20,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:20,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:20,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:20,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:20,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:20,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:20,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:20,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:20,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:20,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:20,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:20,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:41:20,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 18:41:20,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:41:20,884 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 18:41:20,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:41:20,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:20,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:41:20,944 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 18:41:20,944 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 18:41:20,947 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 18:41:21,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 18:41:21,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 18:41:21,342 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 18:41:21,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 18:41:21,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 18:41:21,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 18:41:21,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 18:41:21,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 18:41:21,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 18:41:21,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 18:41:21,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 18:41:21,461 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 18:41:21,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 18:41:21,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:21,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:21,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:21,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 18:41:21,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:21,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:21,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:21,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 18:41:21,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:21,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:21,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:21,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 18:41:21,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:21,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:21,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:21,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 18:41:21,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:22,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:22,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:22,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 18:41:22,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:22,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:22,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:22,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 18:41:22,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:22,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 18:41:22,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:22,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 18:41:22,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 18:41:22,249 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 18:41:22,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 18:41:22,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 18:41:22,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 18:41:22,285 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 18:41:22,285 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 18:41:22,285 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639619646.67 2021-12-15 18:41:22,286 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 40 2021-12-15 18:41:22,319 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639622482.29 2021-12-15 18:41:22,321 | infra | INFO | pool-3-thread-27 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639622482.29' started 2021-12-15 18:41:22,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 18:41:22,328 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 18:41:22,349 | test | DEBUG | pool-3-thread-27 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-15 18:41:22,349 | test | DEBUG | pool-3-thread-27 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-15 18:41:22,403 | test | DEBUG | pool-3-thread-27 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 18:41:22,407 | test | INFO | pool-3-thread-27 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 8.45279011932 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.316759054724 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 3.06136661188 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 1.10468832504 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-15 18:41:22,417 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:41:22,417 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:41:22,436 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:41:22,440 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:41:22,440 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:41:32,454 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:41:32,456 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:41:32,476 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:41:32,483 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:41:32,483 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:41:42,496 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:41:42,496 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:41:42,515 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:41:42,516 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:41:42,517 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:41:52,526 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:41:52,528 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:41:52,546 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:41:52,551 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:41:52,552 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:02,565 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:42:02,565 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:42:02,581 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:02,582 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:02,582 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:12,594 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 18:42:12,594 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 18:42:12,608 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:12,611 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:12,611 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:22,622 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.29 % 2021-12-15 18:42:22,622 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 0.29 2021-12-15 18:42:22,638 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:22,641 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:22,641 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:32,654 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 18:42:32,654 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 18:42:32,670 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:32,671 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:32,671 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:42,680 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.88 % 2021-12-15 18:42:42,681 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 1.88 2021-12-15 18:42:42,697 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:42,698 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:42,700 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:42:52,710 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:42:52,710 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-15 18:42:52,727 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:42:52,730 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:42:52,730 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:02,743 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:02,743 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-15 18:43:02,759 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:02,760 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:02,762 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:12,770 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:12,772 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-15 18:43:12,786 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:12,789 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:12,789 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:22,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:22,336 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 2.62 2021-12-15 18:43:22,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:22,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:43:22,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:22,799 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:22,799 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-15 18:43:22,815 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:22,816 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:22,816 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:32,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:32,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:43:32,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:32,826 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-15 18:43:32,826 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-15 18:43:32,845 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:32,846 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:32,846 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:42,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.14 % 2021-12-15 18:43:42,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:43:42,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:42,857 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.14 % 2021-12-15 18:43:42,857 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 3.14 2021-12-15 18:43:42,871 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:42,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:42,874 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:52,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:43:52,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:43:52,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:43:52,882 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:43:52,884 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-15 18:43:52,903 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:43:52,904 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:43:52,904 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:02,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:02,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:02,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:02,914 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:02,915 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-15 18:44:02,931 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:02,934 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:02,934 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:12,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:12,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:12,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:12,944 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:12,944 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-15 18:44:12,966 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:12,967 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:12,967 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:22,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:22,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:22,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:22,979 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:22,979 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-15 18:44:22,997 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:23,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:23,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:32,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:32,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:32,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:33,013 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-15 18:44:33,013 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-15 18:44:33,029 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:33,032 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:33,032 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:42,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.57 % 2021-12-15 18:44:42,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:42,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:43,042 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.57 % 2021-12-15 18:44:43,042 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.57 2021-12-15 18:44:43,059 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:43,062 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:43,062 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:52,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 18:44:52,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:44:52,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:44:53,072 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.89 % 2021-12-15 18:44:53,072 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 4.89 2021-12-15 18:44:53,088 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:44:53,092 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:44:53,092 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:02,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:02,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:02,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:03,102 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:03,102 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 5.49 2021-12-15 18:45:03,121 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:03,124 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:03,124 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:12,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:12,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:12,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:13,134 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:13,134 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 5.49 2021-12-15 18:45:13,154 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:13,157 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:13,157 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:22,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:22,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:22,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:23,171 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:23,171 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 5.49 2021-12-15 18:45:23,187 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:23,191 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:23,191 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:32,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:32,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:32,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:33,201 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.49 % 2021-12-15 18:45:33,201 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 5.49 2021-12-15 18:45:33,217 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:33,220 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:33,220 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:42,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.01 % 2021-12-15 18:45:42,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:42,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:43,230 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.01 % 2021-12-15 18:45:43,230 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 6.01 2021-12-15 18:45:43,247 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:43,252 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:43,252 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:52,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 18:45:52,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:45:52,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:45:53,263 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.32 % 2021-12-15 18:45:53,263 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 6.32 2021-12-15 18:45:53,282 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:45:53,285 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:45:53,285 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:02,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 18:46:02,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:02,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:03,293 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 18:46:03,293 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 6.93 2021-12-15 18:46:03,309 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:03,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:03,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:12,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.22 % 2021-12-15 18:46:12,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:12,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:13,322 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.22 % 2021-12-15 18:46:13,322 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 7.22 2021-12-15 18:46:13,336 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:13,339 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:13,339 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:22,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.69 % 2021-12-15 18:46:22,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:22,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:23,348 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.69 % 2021-12-15 18:46:23,348 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 7.69 2021-12-15 18:46:23,364 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:23,365 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:23,365 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:32,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.69 % 2021-12-15 18:46:32,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:32,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:33,375 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.69 % 2021-12-15 18:46:33,375 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 7.69 2021-12-15 18:46:33,391 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:33,394 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:33,394 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:42,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 18:46:42,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:42,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:43,404 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 18:46:43,404 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 8.21 2021-12-15 18:46:43,438 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:43,441 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:43,441 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:52,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 18:46:52,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:46:52,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:46:53,450 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 18:46:53,450 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 8.21 2021-12-15 18:46:53,467 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:46:53,469 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:46:53,470 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:02,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-15 18:47:02,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:02,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:03,480 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.28 % 2021-12-15 18:47:03,480 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 9.28 2021-12-15 18:47:03,496 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:03,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:03,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:12,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.59 % 2021-12-15 18:47:12,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:12,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:13,512 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.59 % 2021-12-15 18:47:13,512 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 9.59 2021-12-15 18:47:13,529 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:13,532 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:13,532 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:22,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.59 % 2021-12-15 18:47:22,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:22,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:23,558 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.59 % 2021-12-15 18:47:23,558 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 9.59 2021-12-15 18:47:23,578 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:23,581 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:23,581 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:32,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.02 % 2021-12-15 18:47:32,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:32,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:33,592 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.02 % 2021-12-15 18:47:33,592 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 10.02 2021-12-15 18:47:33,608 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:33,611 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:33,611 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:42,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:47:42,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:42,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:43,621 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:47:43,622 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 10.54 2021-12-15 18:47:43,640 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:43,642 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:43,642 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:52,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:47:52,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:47:52,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:47:53,654 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:47:53,655 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 10.54 2021-12-15 18:47:53,677 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:47:53,680 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:47:53,680 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:02,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:48:02,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:02,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:03,691 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-15 18:48:03,691 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 10.54 2021-12-15 18:48:03,710 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:03,711 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:03,711 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:12,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:12,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:12,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:13,721 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:13,721 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.46 2021-12-15 18:48:13,736 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:13,737 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:13,737 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:22,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:22,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:22,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:23,746 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:23,747 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.46 2021-12-15 18:48:23,763 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:23,765 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:23,765 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:32,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:32,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:32,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:33,779 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:33,779 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.46 2021-12-15 18:48:33,796 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:33,798 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:33,798 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:42,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:42,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:42,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:43,806 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.46 % 2021-12-15 18:48:43,808 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.46 2021-12-15 18:48:43,826 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:43,828 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:43,828 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:52,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 18:48:52,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:48:52,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:48:53,841 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 18:48:53,841 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.97 2021-12-15 18:48:53,862 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:48:53,865 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:48:53,865 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:02,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 18:49:02,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:02,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:03,877 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 18:49:03,877 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 11.97 2021-12-15 18:49:03,900 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:03,903 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:03,903 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:12,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.57 % 2021-12-15 18:49:12,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:12,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:13,914 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.57 % 2021-12-15 18:49:13,914 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 12.57 2021-12-15 18:49:13,928 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:13,930 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:13,930 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:23,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:23,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:23,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:23,943 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:23,944 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 18:49:23,966 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:23,967 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:23,967 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:33,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:33,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:33,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:33,977 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:33,977 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 18:49:33,993 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:33,996 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:33,996 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:43,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:43,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:43,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:44,007 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:44,007 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 18:49:44,026 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:44,029 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:44,029 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:53,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:53,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:49:53,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:49:54,042 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.89 % 2021-12-15 18:49:54,043 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 12.89 2021-12-15 18:49:54,062 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:49:54,063 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:49:54,065 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:03,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.18 % 2021-12-15 18:50:03,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:03,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:04,075 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.18 % 2021-12-15 18:50:04,075 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 13.18 2021-12-15 18:50:04,092 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:04,095 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:04,095 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:13,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.79 % 2021-12-15 18:50:13,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:13,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:14,105 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.30 % 2021-12-15 18:50:14,105 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.3 2021-12-15 18:50:14,122 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:14,125 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:14,125 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:23,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:23,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:23,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:24,137 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:24,137 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.62 2021-12-15 18:50:24,155 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:24,158 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:24,158 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:33,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:33,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:33,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:34,168 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:34,168 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.62 2021-12-15 18:50:34,184 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:34,187 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:34,187 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:43,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:43,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:43,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:44,197 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:44,197 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.62 2021-12-15 18:50:44,216 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:44,219 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:44,219 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:53,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:53,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:50:53,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:50:54,229 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:50:54,230 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.62 2021-12-15 18:50:54,259 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:50:54,262 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:50:54,262 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:03,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:51:03,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:03,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:04,272 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.62 % 2021-12-15 18:51:04,272 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 14.62 2021-12-15 18:51:04,289 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:04,292 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:04,292 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:13,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.40 % 2021-12-15 18:51:13,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:13,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:14,302 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.40 % 2021-12-15 18:51:14,302 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 15.4 2021-12-15 18:51:14,319 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:14,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:14,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:23,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.92 % 2021-12-15 18:51:23,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:23,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:24,332 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.92 % 2021-12-15 18:51:24,332 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 15.92 2021-12-15 18:51:24,348 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:24,351 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:24,351 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:33,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:33,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:33,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:34,361 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:34,374 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 16.52 2021-12-15 18:51:34,391 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:34,392 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:34,394 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:43,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:43,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:43,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:44,403 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:44,403 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 16.52 2021-12-15 18:51:44,417 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:44,420 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:44,420 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:53,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:53,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:51:53,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:51:54,431 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.52 % 2021-12-15 18:51:54,431 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 16.52 2021-12-15 18:51:54,450 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:51:54,453 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:51:54,453 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:03,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 18:52:03,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:03,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:04,461 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 18:52:04,463 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 18:52:04,479 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:04,480 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:04,482 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:13,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 18:52:13,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:13,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:14,492 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.99 % 2021-12-15 18:52:14,492 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 16.99 2021-12-15 18:52:14,507 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:14,509 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:14,509 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:23,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.51 % 2021-12-15 18:52:23,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:23,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:24,519 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.51 % 2021-12-15 18:52:24,519 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 17.51 2021-12-15 18:52:24,535 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:24,538 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:24,538 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:33,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.54 % 2021-12-15 18:52:33,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:33,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:34,548 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.54 % 2021-12-15 18:52:34,548 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.54 2021-12-15 18:52:34,563 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:34,566 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:34,566 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:43,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:52:43,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:43,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:44,581 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:52:44,581 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.85 2021-12-15 18:52:44,595 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:44,598 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:44,598 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:53,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:52:53,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:52:53,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:52:54,608 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:52:54,608 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.85 2021-12-15 18:52:54,625 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:52:54,627 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:52:54,627 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:03,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:03,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:03,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:04,638 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:04,638 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.85 2021-12-15 18:53:04,655 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:04,658 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:04,658 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:13,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:13,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:13,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:14,668 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:14,668 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.85 2021-12-15 18:53:14,684 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:14,687 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:14,687 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:23,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:23,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:23,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:24,697 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.85 % 2021-12-15 18:53:24,697 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 18.85 2021-12-15 18:53:24,736 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:24,739 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:24,739 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:33,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.97 % 2021-12-15 18:53:33,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:33,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:34,747 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.97 % 2021-12-15 18:53:34,749 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 19.97 2021-12-15 18:53:34,763 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:34,765 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:34,766 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:43,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.29 % 2021-12-15 18:53:43,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:43,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:44,776 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.29 % 2021-12-15 18:53:44,776 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 20.29 2021-12-15 18:53:44,795 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:44,796 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:44,796 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:53,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:53:53,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:53:53,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:53:54,808 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:53:54,809 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 20.58 2021-12-15 18:53:54,825 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:53:54,828 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:53:54,828 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:03,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:03,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:03,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:04,838 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:04,838 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 20.58 2021-12-15 18:54:04,852 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:04,855 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:04,855 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:13,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:13,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:13,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:14,865 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:14,865 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 20.58 2021-12-15 18:54:14,881 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:14,882 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:14,882 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:23,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:23,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:23,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:24,891 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.58 % 2021-12-15 18:54:24,892 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 20.58 2021-12-15 18:54:24,907 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:24,910 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:24,910 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:33,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.18 % 2021-12-15 18:54:33,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:33,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:34,921 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.18 % 2021-12-15 18:54:34,921 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 21.18 2021-12-15 18:54:34,937 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:34,938 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:34,940 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:43,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.70 % 2021-12-15 18:54:43,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:43,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:44,950 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.70 % 2021-12-15 18:54:44,950 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 21.7 2021-12-15 18:54:44,970 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:44,973 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:44,973 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:53,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.70 % 2021-12-15 18:54:53,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:54:53,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:54:54,983 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.70 % 2021-12-15 18:54:54,983 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 21.7 2021-12-15 18:54:54,999 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:54:55,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:54:55,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:03,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:03,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:03,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:05,012 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:05,012 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 22.01 2021-12-15 18:55:05,036 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:05,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:05,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:13,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:13,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:13,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:15,051 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:15,051 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 22.01 2021-12-15 18:55:15,065 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:15,068 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:15,069 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:23,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:23,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:23,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:25,078 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-15 18:55:25,078 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 22.01 2021-12-15 18:55:25,098 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:25,101 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:25,101 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:33,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.48 % 2021-12-15 18:55:33,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:33,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:35,109 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.48 % 2021-12-15 18:55:35,109 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 22.48 2021-12-15 18:55:35,125 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:35,127 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:35,127 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:43,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.48 % 2021-12-15 18:55:43,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:43,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:45,137 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-15 18:55:45,137 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.09 2021-12-15 18:55:45,153 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:45,155 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:45,155 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:53,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:55:53,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:55:53,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:55:55,164 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:55:55,165 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.6 2021-12-15 18:55:55,180 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:55:55,184 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:55:55,184 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:03,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:03,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:03,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:05,194 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:05,194 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.6 2021-12-15 18:56:05,210 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:05,213 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:05,213 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:13,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:13,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:13,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:15,223 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:15,224 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.6 2021-12-15 18:56:15,240 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:15,243 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:15,243 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:23,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:23,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:23,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:25,255 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.60 % 2021-12-15 18:56:25,256 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.6 2021-12-15 18:56:25,273 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:25,276 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:25,276 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:33,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.92 % 2021-12-15 18:56:33,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:33,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:35,288 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.92 % 2021-12-15 18:56:35,288 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 23.92 2021-12-15 18:56:35,303 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:35,306 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:35,306 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:43,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-15 18:56:43,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:43,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:45,318 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-15 18:56:45,319 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-15 18:56:45,335 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:45,338 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:45,338 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:53,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-15 18:56:53,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:56:53,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:56:55,349 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-15 18:56:55,349 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-15 18:56:55,368 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:56:55,371 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:56:55,371 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:03,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:03,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:03,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:05,382 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:05,382 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 25.46 2021-12-15 18:57:05,400 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:05,403 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:05,403 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:13,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:13,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:13,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:15,413 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:15,414 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 25.46 2021-12-15 18:57:15,428 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:15,430 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:15,430 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:23,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:23,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:23,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:25,440 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.46 % 2021-12-15 18:57:25,441 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 25.46 2021-12-15 18:57:25,456 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:25,459 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:25,459 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:33,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.78 % 2021-12-15 18:57:33,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:33,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:35,471 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.07 % 2021-12-15 18:57:35,471 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 26.07 2021-12-15 18:57:35,489 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:35,492 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:35,492 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:43,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.07 % 2021-12-15 18:57:43,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:43,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:45,503 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.07 % 2021-12-15 18:57:45,503 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 26.07 2021-12-15 18:57:45,519 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:45,522 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:45,523 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:53,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:57:53,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:57:53,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:57:55,536 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:57:55,536 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 26.59 2021-12-15 18:57:55,553 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:57:55,555 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:57:55,556 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:04,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:58:04,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:04,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:05,569 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:58:05,569 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 26.59 2021-12-15 18:58:05,585 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:05,586 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:05,588 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:14,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:58:14,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:14,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:15,599 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.59 % 2021-12-15 18:58:15,599 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 26.59 2021-12-15 18:58:15,615 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:15,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:15,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:24,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:24,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:24,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:25,631 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:25,631 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 27.37 2021-12-15 18:58:25,645 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:25,648 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:25,648 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:34,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:34,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:34,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:35,660 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:35,660 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 27.37 2021-12-15 18:58:35,674 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:35,677 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:35,677 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:44,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:44,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:44,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:45,687 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:45,687 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 27.37 2021-12-15 18:58:45,703 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:45,703 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:45,704 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:54,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:54,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:58:54,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:58:55,716 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.37 % 2021-12-15 18:58:55,716 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 27.37 2021-12-15 18:58:55,732 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:58:55,733 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:58:55,733 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:04,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:04,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:04,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:05,747 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:05,747 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.2 2021-12-15 18:59:05,767 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:05,770 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:05,770 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:14,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:14,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:14,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:15,782 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:15,782 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.2 2021-12-15 18:59:15,801 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:15,803 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:15,805 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:24,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:24,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:24,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:25,816 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:25,816 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.2 2021-12-15 18:59:25,835 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:25,838 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:25,838 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:34,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:34,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:34,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:35,851 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:35,851 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.2 2021-12-15 18:59:35,868 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:35,869 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:35,869 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:44,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:44,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:44,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:45,882 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.20 % 2021-12-15 18:59:45,882 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.2 2021-12-15 18:59:45,900 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:45,903 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:45,903 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:54,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.52 % 2021-12-15 18:59:54,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 18:59:54,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 18:59:55,914 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.52 % 2021-12-15 18:59:55,914 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 28.52 2021-12-15 18:59:55,928 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 18:59:55,930 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 18:59:55,930 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:04,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:04,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:04,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:05,941 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:05,941 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.04 2021-12-15 19:00:05,957 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:05,959 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:05,959 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:14,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:14,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:14,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:15,970 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:15,970 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.04 2021-12-15 19:00:15,992 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:15,993 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:15,993 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:24,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:24,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:24,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:26,005 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.04 % 2021-12-15 19:00:26,006 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.04 2021-12-15 19:00:26,022 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:26,023 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:26,023 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:34,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.35 % 2021-12-15 19:00:34,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:34,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:36,036 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.35 % 2021-12-15 19:00:36,036 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.35 2021-12-15 19:00:36,056 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:36,059 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:36,059 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:44,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 19:00:44,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:44,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:46,069 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 19:00:46,071 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 19:00:46,088 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:46,089 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:46,089 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:54,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 19:00:54,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:00:54,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:00:56,101 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 19:00:56,101 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 19:00:56,119 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:00:56,121 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:00:56,122 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:04,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.34 % 2021-12-15 19:01:04,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:04,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:06,134 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.34 % 2021-12-15 19:01:06,134 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.34 2021-12-15 19:01:06,150 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:06,151 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:06,151 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:14,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.34 % 2021-12-15 19:01:14,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:14,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:16,163 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:16,163 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.65 2021-12-15 19:01:16,177 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:16,180 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:16,180 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:24,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:24,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:24,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:26,191 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:26,191 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.65 2021-12-15 19:01:26,209 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:26,210 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:26,210 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:34,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:34,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:34,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:36,221 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:36,221 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.65 2021-12-15 19:01:36,249 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:36,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:36,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:44,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:44,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:44,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:46,273 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:46,273 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.65 2021-12-15 19:01:46,295 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:46,299 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:46,299 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:54,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:54,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:01:54,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:01:56,312 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.65 % 2021-12-15 19:01:56,312 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 30.65 2021-12-15 19:01:56,328 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:01:56,331 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:01:56,332 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:04,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:04,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:04,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:06,342 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:06,342 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 31.78 2021-12-15 19:02:06,358 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:06,361 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:06,361 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:14,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:14,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:14,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:16,375 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:16,377 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 31.78 2021-12-15 19:02:16,395 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:16,398 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:16,398 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:24,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:24,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:24,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:26,418 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:26,418 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 31.78 2021-12-15 19:02:26,436 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:26,438 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:26,438 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:34,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:34,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:34,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:36,450 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.78 % 2021-12-15 19:02:36,450 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 31.78 2021-12-15 19:02:36,467 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:36,470 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:36,470 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:44,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.10 % 2021-12-15 19:02:44,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:44,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:46,486 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.10 % 2021-12-15 19:02:46,486 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.1 2021-12-15 19:02:46,500 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:46,503 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:46,503 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:54,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:02:54,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:02:54,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:02:56,520 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:02:56,522 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.57 2021-12-15 19:02:56,551 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:02:56,552 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:02:56,552 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:04,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:03:04,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:04,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:06,563 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:03:06,563 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.57 2021-12-15 19:03:06,588 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:06,591 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:06,591 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:14,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:03:14,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:14,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:16,602 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:03:16,602 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.57 2021-12-15 19:03:16,619 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:16,621 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:16,621 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:24,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.57 % 2021-12-15 19:03:24,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:24,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:26,634 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:26,634 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 19:03:26,653 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:26,654 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:26,654 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:34,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:34,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:34,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:36,665 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:36,665 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 19:03:36,681 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:36,684 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:36,684 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:44,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:44,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:44,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:46,694 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:46,696 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 19:03:46,711 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:46,713 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:46,713 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:54,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:54,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:03:54,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:03:56,724 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:03:56,724 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 19:03:56,740 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:03:56,743 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:03:56,743 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:04,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:04:04,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:04,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:06,753 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-15 19:04:06,753 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-15 19:04:06,770 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:06,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:06,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:14,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:14,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:14,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:16,789 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:16,789 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 19:04:16,809 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:16,813 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:16,813 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:24,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:24,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:24,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:26,828 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:26,828 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 19:04:26,845 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:26,849 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:26,849 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:34,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:34,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:34,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:36,861 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:36,861 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 19:04:36,878 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:36,882 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:36,882 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:44,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:44,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:44,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:46,898 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:46,898 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.2 2021-12-15 19:04:46,918 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:46,923 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:46,923 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:54,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.20 % 2021-12-15 19:04:54,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:04:54,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:04:56,934 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.51 % 2021-12-15 19:04:56,934 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.51 2021-12-15 19:04:56,953 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:04:56,954 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:04:56,954 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:04,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.51 % 2021-12-15 19:05:04,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:04,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:06,966 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.51 % 2021-12-15 19:05:06,967 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.51 2021-12-15 19:05:06,983 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:06,983 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:06,984 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:14,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.51 % 2021-12-15 19:05:14,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:14,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:16,996 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.51 % 2021-12-15 19:05:16,996 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.51 2021-12-15 19:05:17,013 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:17,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:17,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:24,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.98 % 2021-12-15 19:05:24,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:24,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:27,028 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.98 % 2021-12-15 19:05:27,028 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.98 2021-12-15 19:05:27,046 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:27,048 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:27,048 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:34,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.98 % 2021-12-15 19:05:34,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:34,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:37,059 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.98 % 2021-12-15 19:05:37,059 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 33.98 2021-12-15 19:05:37,075 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:37,076 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:37,076 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:44,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:05:44,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:44,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:47,088 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:05:47,088 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.3 2021-12-15 19:05:47,108 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:47,109 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:47,111 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:54,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:05:54,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:05:54,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:05:57,121 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:05:57,121 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.3 2021-12-15 19:05:57,137 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:05:57,138 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:05:57,140 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:04,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:06:04,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:04,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:07,151 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:06:07,153 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.3 2021-12-15 19:06:07,170 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:07,171 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:07,171 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:14,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:06:14,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:14,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:17,183 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.30 % 2021-12-15 19:06:17,184 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.3 2021-12-15 19:06:17,200 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:17,201 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:17,201 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:24,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.59 % 2021-12-15 19:06:25,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:25,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:27,213 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.59 % 2021-12-15 19:06:27,213 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.59 2021-12-15 19:06:27,230 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:27,232 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:27,232 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:35,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 19:06:35,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:35,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:37,243 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.91 % 2021-12-15 19:06:37,243 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 34.91 2021-12-15 19:06:37,257 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:37,259 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:37,259 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:45,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:06:45,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:45,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:47,270 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:06:47,272 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.2 2021-12-15 19:06:47,286 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:47,289 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:47,289 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:55,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:06:55,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:06:55,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:06:57,301 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:06:57,301 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.2 2021-12-15 19:06:57,316 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:06:57,319 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:06:57,319 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:05,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:07:05,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:05,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:07,332 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.20 % 2021-12-15 19:07:07,332 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.2 2021-12-15 19:07:07,351 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:07,355 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:07,355 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:15,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:15,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:15,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:17,367 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:17,367 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.99 2021-12-15 19:07:17,384 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:17,388 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:17,388 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:25,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:25,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:25,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:27,400 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:27,400 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.99 2021-12-15 19:07:27,421 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:27,424 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:27,424 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:35,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:35,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:35,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:37,438 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:37,438 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.99 2021-12-15 19:07:37,460 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:37,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:37,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:45,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:45,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:45,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:47,476 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:47,476 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.99 2021-12-15 19:07:47,493 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:47,497 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:47,497 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:55,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:55,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:07:55,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:07:57,509 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.99 % 2021-12-15 19:07:57,509 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 35.99 2021-12-15 19:07:57,526 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:07:57,529 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:07:57,529 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:05,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:05,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:05,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:07,539 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:07,539 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.31 2021-12-15 19:08:07,556 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:07,559 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:07,559 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:15,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:15,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:15,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:17,569 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:17,569 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.31 2021-12-15 19:08:17,586 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:17,589 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:17,589 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:25,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:25,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:25,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:27,601 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:27,601 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.31 2021-12-15 19:08:27,615 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:27,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:27,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:35,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:35,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:35,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:37,631 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-15 19:08:37,631 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.31 2021-12-15 19:08:37,650 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:37,653 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:37,653 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:45,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:08:45,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:45,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:47,673 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:08:47,673 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.62 2021-12-15 19:08:47,688 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:47,690 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:47,690 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:55,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:08:55,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:08:55,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:08:57,700 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:08:57,701 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.62 2021-12-15 19:08:57,720 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:08:57,721 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:08:57,721 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:05,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:05,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:05,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:07,733 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:07,733 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.62 2021-12-15 19:09:07,747 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:07,750 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:07,750 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:15,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:15,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:15,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:17,762 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:17,762 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.62 2021-12-15 19:09:17,779 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:17,780 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:17,780 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:25,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:25,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:25,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:27,792 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.62 % 2021-12-15 19:09:27,792 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.62 2021-12-15 19:09:27,812 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:27,815 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:27,815 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:35,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.94 % 2021-12-15 19:09:35,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:35,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:37,828 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.94 % 2021-12-15 19:09:37,828 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 36.94 2021-12-15 19:09:37,858 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:37,861 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:37,862 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:45,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:09:45,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:45,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:47,875 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:09:47,875 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.41 2021-12-15 19:09:47,892 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:47,895 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:47,895 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:55,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:09:55,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:09:55,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:09:57,907 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:09:57,907 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.41 2021-12-15 19:09:57,927 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:09:57,930 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:09:57,931 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:05,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:10:05,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:05,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:07,943 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:10:07,943 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.41 2021-12-15 19:10:07,959 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:07,961 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:07,961 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:15,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.41 % 2021-12-15 19:10:15,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:15,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:17,973 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:17,973 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.72 2021-12-15 19:10:17,987 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:17,990 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:17,990 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:25,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:25,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:25,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:28,003 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:28,003 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.72 2021-12-15 19:10:28,023 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:28,025 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:28,026 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:35,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:35,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:35,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:38,038 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:38,038 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.72 2021-12-15 19:10:38,058 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:38,059 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:38,059 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:45,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:45,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:45,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:48,069 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.72 % 2021-12-15 19:10:48,069 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 37.72 2021-12-15 19:10:48,085 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:48,088 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:48,088 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:55,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.02 % 2021-12-15 19:10:55,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:10:55,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:10:58,099 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.02 % 2021-12-15 19:10:58,101 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 38.02 2021-12-15 19:10:58,115 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:10:58,117 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:10:58,117 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:05,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.63 % 2021-12-15 19:11:05,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:05,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:08,128 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.63 % 2021-12-15 19:11:08,130 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 38.63 2021-12-15 19:11:08,147 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:08,148 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:08,150 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:15,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.63 % 2021-12-15 19:11:15,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:15,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:18,161 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.63 % 2021-12-15 19:11:18,161 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 38.63 2021-12-15 19:11:18,178 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:18,180 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:18,181 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:25,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.63 % 2021-12-15 19:11:25,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:25,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:28,191 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-15 19:11:28,193 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.1 2021-12-15 19:11:28,209 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:28,210 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:28,210 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:35,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-15 19:11:35,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:35,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:38,221 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-15 19:11:38,221 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.1 2021-12-15 19:11:38,239 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:38,240 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:38,240 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:45,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-15 19:11:45,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:45,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:48,252 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.41 % 2021-12-15 19:11:48,252 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.41 2021-12-15 19:11:48,266 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:48,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:48,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:55,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.41 % 2021-12-15 19:11:55,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:11:55,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:11:58,279 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.41 % 2021-12-15 19:11:58,279 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.41 2021-12-15 19:11:58,295 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:11:58,296 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:11:58,296 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:05,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.41 % 2021-12-15 19:12:05,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:05,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:08,306 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.41 % 2021-12-15 19:12:08,308 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.41 2021-12-15 19:12:08,322 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:08,325 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:08,325 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:15,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:15,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:15,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:18,335 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:18,335 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.73 2021-12-15 19:12:18,351 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:18,354 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:18,354 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:25,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:25,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:25,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:28,365 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:28,365 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.73 2021-12-15 19:12:28,382 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:28,385 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:28,385 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:35,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:35,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:35,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:38,398 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:38,398 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.73 2021-12-15 19:12:38,414 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:38,417 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:38,417 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:45,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:45,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:45,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:48,430 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.73 % 2021-12-15 19:12:48,430 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 39.73 2021-12-15 19:12:48,446 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:48,448 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:48,450 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:55,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:12:55,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:12:55,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:12:58,463 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:12:58,463 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 40.2 2021-12-15 19:12:58,480 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:12:58,483 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:12:58,483 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:05,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:13:05,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:05,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:08,493 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:13:08,493 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 40.2 2021-12-15 19:13:08,509 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:08,513 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:08,513 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:15,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:13:15,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:15,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:18,526 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.20 % 2021-12-15 19:13:18,526 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 40.2 2021-12-15 19:13:18,549 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:18,559 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:18,559 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:25,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.79 % 2021-12-15 19:13:25,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:25,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:28,585 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.79 % 2021-12-15 19:13:28,585 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 40.79 2021-12-15 19:13:28,601 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:28,605 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:28,605 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:35,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:35,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:35,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:38,615 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:38,615 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.26 2021-12-15 19:13:38,631 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:38,634 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:38,634 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:45,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:45,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:45,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:48,644 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:48,644 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.26 2021-12-15 19:13:48,661 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:48,663 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:48,663 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:55,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:55,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:13:55,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:13:58,674 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:13:58,674 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.26 2021-12-15 19:13:58,690 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:13:58,693 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:13:58,694 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:05,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:14:05,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:05,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:08,703 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:14:08,703 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.26 2021-12-15 19:14:08,720 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:08,721 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:08,721 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:15,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:14:15,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:15,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:18,732 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:14:18,733 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.26 2021-12-15 19:14:18,747 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:18,750 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:18,750 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:25,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.26 % 2021-12-15 19:14:25,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:25,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:28,763 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.69 % 2021-12-15 19:14:28,763 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.69 2021-12-15 19:14:28,782 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:28,785 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:28,786 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:36,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.98 % 2021-12-15 19:14:36,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:36,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:38,795 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.98 % 2021-12-15 19:14:38,795 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 41.98 2021-12-15 19:14:38,809 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:38,812 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:38,812 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:46,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:14:46,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:46,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:48,822 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:14:48,822 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:14:48,841 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:48,842 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:48,842 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:56,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:14:56,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:14:56,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:14:58,854 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:14:58,854 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:14:58,869 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:14:58,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:14:58,872 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:06,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:06,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:06,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:08,882 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:08,882 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:15:08,897 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:08,900 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:08,901 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:16,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:16,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:16,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:18,914 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:18,914 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:15:18,930 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:18,933 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:18,934 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:26,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:26,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:26,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:28,946 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:28,946 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:15:28,963 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:28,966 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:28,967 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:36,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:36,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:36,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:38,977 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:38,977 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 42.88 2021-12-15 19:15:38,994 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:38,997 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:38,997 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:46,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.88 % 2021-12-15 19:15:46,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:46,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:49,007 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.30 % 2021-12-15 19:15:49,007 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 43.3 2021-12-15 19:15:49,023 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:49,026 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:49,026 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:56,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.30 % 2021-12-15 19:15:56,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:15:56,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:15:59,036 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.30 % 2021-12-15 19:15:59,036 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 43.3 2021-12-15 19:15:59,052 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:15:59,055 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:15:59,055 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:06,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.30 % 2021-12-15 19:16:06,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:06,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:09,078 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.60 % 2021-12-15 19:16:09,078 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 43.6 2021-12-15 19:16:09,095 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:09,098 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:09,098 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:16,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.07 % 2021-12-15 19:16:16,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:16,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:19,111 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:19,111 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.54 2021-12-15 19:16:19,131 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:19,132 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:19,134 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:26,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:26,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:26,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:29,144 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:29,144 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.54 2021-12-15 19:16:29,158 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:29,161 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:29,161 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:36,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:36,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:36,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:39,171 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:39,171 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.54 2021-12-15 19:16:39,187 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:39,188 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:39,190 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:46,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:46,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:46,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:49,200 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:49,200 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.54 2021-12-15 19:16:49,220 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:49,223 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:49,223 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:56,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:56,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:16:56,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:16:59,233 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:16:59,233 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.54 2021-12-15 19:16:59,250 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:16:59,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:16:59,253 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:06,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.54 % 2021-12-15 19:17:06,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:06,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:09,263 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-15 19:17:09,263 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.96 2021-12-15 19:17:09,279 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:09,282 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:09,282 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:16,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-15 19:17:16,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:16,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:19,293 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-15 19:17:19,295 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 44.96 2021-12-15 19:17:19,309 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:19,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:19,312 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:26,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.39 % 2021-12-15 19:17:26,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:26,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:29,322 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.68 % 2021-12-15 19:17:29,322 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 45.68 2021-12-15 19:17:29,338 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:29,339 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:29,339 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:36,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.68 % 2021-12-15 19:17:36,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:36,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:39,355 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:17:39,355 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.15 2021-12-15 19:17:39,374 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:39,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:39,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:46,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:17:46,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:46,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:49,385 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:17:49,387 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.15 2021-12-15 19:17:49,403 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:49,404 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:49,404 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:56,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:17:56,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:17:56,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:17:59,414 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:17:59,414 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.15 2021-12-15 19:17:59,431 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:17:59,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:17:59,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:06,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:18:06,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:06,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:09,459 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:18:09,460 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.15 2021-12-15 19:18:09,474 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:09,477 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:09,479 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:16,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:18:16,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:16,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:19,490 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:18:19,490 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.15 2021-12-15 19:18:19,507 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:19,510 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:19,510 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:26,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.15 % 2021-12-15 19:18:26,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:26,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:29,522 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:29,522 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.58 2021-12-15 19:18:29,540 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:29,543 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:29,543 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:36,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:36,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:36,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:39,569 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:39,569 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.58 2021-12-15 19:18:39,585 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:39,588 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:39,588 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:46,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:46,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:46,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:49,598 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:49,598 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.58 2021-12-15 19:18:49,614 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:49,617 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:49,618 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:56,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:56,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:18:56,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:18:59,628 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.58 % 2021-12-15 19:18:59,628 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 46.58 2021-12-15 19:18:59,644 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:18:59,648 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:18:59,648 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:06,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:06,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:06,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:09,660 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:09,660 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.05 2021-12-15 19:19:09,677 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:09,680 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:09,680 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:16,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:16,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:16,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:19,690 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:19,690 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.05 2021-12-15 19:19:19,707 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:19,711 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:19,711 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:26,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:26,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:26,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:29,723 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:29,723 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.05 2021-12-15 19:19:29,739 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:29,742 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:29,742 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:36,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-15 19:19:36,561 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:36,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:39,753 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:19:39,753 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.35 2021-12-15 19:19:39,769 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:39,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:39,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:46,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:19:46,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:46,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:49,783 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:19:49,785 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.35 2021-12-15 19:19:49,801 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:49,802 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:49,802 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:56,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:19:56,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:19:56,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:19:59,813 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:19:59,815 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.35 2021-12-15 19:19:59,829 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:19:59,832 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:19:59,832 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:06,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:20:06,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:06,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:09,842 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:20:09,842 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.35 2021-12-15 19:20:09,858 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:20:09,859 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:20:09,859 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:16,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:20:16,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:16,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:19,869 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:20:19,869 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.35 2021-12-15 19:20:19,891 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:20:19,894 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:20:19,894 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:26,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.35 % 2021-12-15 19:20:26,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:26,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:29,905 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:29,905 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:20:29,923 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:20:29,926 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:20:29,926 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:36,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:36,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:36,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:39,937 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:39,937 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:20:39,957 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:20:39,960 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:20:39,960 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:46,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:46,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:46,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:49,973 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:49,973 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:20:49,990 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:20:49,993 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:20:49,993 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:20:56,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:20:56,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:20:56,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:00,016 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:00,016 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:21:00,033 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:00,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:00,039 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:06,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:06,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:06,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:10,051 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:10,052 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:21:10,068 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:10,071 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:10,071 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:16,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:16,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:16,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:20,084 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:20,085 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:21:20,101 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:20,104 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:20,105 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:26,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:26,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:26,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:30,114 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.64 % 2021-12-15 19:21:30,115 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 47.64 2021-12-15 19:21:30,131 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:30,134 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:30,134 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:36,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:21:36,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:36,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:40,144 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:21:40,144 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.07 2021-12-15 19:21:40,161 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:40,163 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:40,163 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:46,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:21:46,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:46,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:50,176 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:21:50,176 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.07 2021-12-15 19:21:50,194 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:21:50,196 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:21:50,196 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:21:56,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:21:56,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:21:56,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:00,207 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:00,207 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.07 2021-12-15 19:22:00,229 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:00,232 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:00,232 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:06,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:06,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:06,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:10,243 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:10,243 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.07 2021-12-15 19:22:10,265 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:10,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:10,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:16,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:16,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:16,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:20,286 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:20,286 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.07 2021-12-15 19:22:20,305 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:20,308 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:20,308 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:26,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.07 % 2021-12-15 19:22:26,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:26,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:30,321 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.36 % 2021-12-15 19:22:30,322 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.36 2021-12-15 19:22:30,338 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:30,341 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:30,341 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:36,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.36 % 2021-12-15 19:22:36,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:36,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:40,351 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:22:40,352 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-15 19:22:40,368 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:40,371 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:40,371 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:46,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:22:46,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:46,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:50,382 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:22:50,382 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-15 19:22:50,400 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:22:50,403 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:22:50,403 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:22:56,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:22:56,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:22:56,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:00,414 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:23:00,414 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-15 19:23:00,428 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:00,431 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:00,431 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:06,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:23:06,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:06,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:10,443 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-15 19:23:10,443 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-15 19:23:10,460 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:10,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:10,463 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:16,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:17,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:17,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:20,474 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:20,476 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.08 2021-12-15 19:23:20,494 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:20,497 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:20,499 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:27,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:27,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:27,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:30,515 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:30,515 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.08 2021-12-15 19:23:30,535 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:30,539 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:30,539 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:37,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:37,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:37,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:40,553 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.08 % 2021-12-15 19:23:40,553 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.08 2021-12-15 19:23:40,572 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:40,575 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:40,575 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:47,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.51 % 2021-12-15 19:23:47,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:47,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:50,588 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.51 % 2021-12-15 19:23:50,588 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.51 2021-12-15 19:23:50,605 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:23:50,607 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:23:50,607 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:23:57,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:23:57,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:23:57,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:00,618 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:00,618 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:00,632 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:00,635 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:00,635 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:07,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:07,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:07,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:10,647 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:10,648 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:10,664 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:10,667 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:10,667 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:17,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:17,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:17,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:20,678 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:20,678 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:20,694 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:20,696 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:20,697 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:27,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:27,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:27,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:30,707 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:30,709 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:30,726 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:30,729 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:30,729 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:37,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:37,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:37,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:40,740 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:40,740 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:40,756 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:40,759 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:40,759 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:47,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:47,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:47,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:50,769 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-15 19:24:50,769 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 49.98 2021-12-15 19:24:50,786 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:24:50,788 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:24:50,788 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:24:57,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.40 % 2021-12-15 19:24:57,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:24:57,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:00,799 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.40 % 2021-12-15 19:25:00,799 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.4 2021-12-15 19:25:00,815 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:00,818 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:00,818 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:07,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.40 % 2021-12-15 19:25:07,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:07,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:10,828 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.40 % 2021-12-15 19:25:10,828 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.4 2021-12-15 19:25:10,844 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:10,846 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:10,846 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:17,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:17,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:17,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:20,857 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:20,858 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:25:20,872 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:20,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:20,875 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:27,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:27,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:27,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:30,885 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:30,885 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:25:30,901 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:30,904 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:30,904 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:37,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:37,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:37,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:40,915 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:40,917 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:25:40,934 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:40,936 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:40,937 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:47,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:47,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:47,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:50,947 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:50,948 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:25:50,967 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:25:50,973 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:25:50,973 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:25:57,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:25:57,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:25:57,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:00,984 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:00,984 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:26:01,003 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:01,006 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:01,006 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:07,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:07,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:07,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:11,019 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:11,019 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:26:11,033 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:11,036 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:11,036 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:17,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:17,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:17,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:21,049 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:21,049 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:26:21,065 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:21,069 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:21,069 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:27,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:27,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:27,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:31,081 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.83 % 2021-12-15 19:26:31,081 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 50.83 2021-12-15 19:26:31,101 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:31,104 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:31,104 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:37,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.26 % 2021-12-15 19:26:37,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:37,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:41,114 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.26 % 2021-12-15 19:26:41,114 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 51.26 2021-12-15 19:26:41,130 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:41,131 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:41,131 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:47,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:26:47,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:47,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:51,144 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:26:51,144 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 51.73 2021-12-15 19:26:51,161 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:26:51,163 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:26:51,163 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:26:57,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:26:57,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:26:57,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:01,174 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:27:01,174 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 51.73 2021-12-15 19:27:01,190 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:01,191 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:01,191 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:07,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:27:07,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:07,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:11,201 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:27:11,201 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 51.73 2021-12-15 19:27:11,217 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:11,217 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:11,219 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:17,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:27:17,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:17,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:21,229 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.73 % 2021-12-15 19:27:21,229 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 51.73 2021-12-15 19:27:21,243 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:21,246 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:21,246 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:27,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:27,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:27,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:31,257 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:31,257 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-15 19:27:31,275 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:31,278 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:31,278 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:37,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:37,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:37,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:41,289 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:41,289 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-15 19:27:41,306 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:41,309 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:41,309 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:47,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:47,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:47,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:51,321 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:51,322 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-15 19:27:51,338 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:27:51,342 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:27:51,342 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:27:57,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-15 19:27:57,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:27:57,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:01,355 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.45 % 2021-12-15 19:28:01,355 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.45 2021-12-15 19:28:01,372 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:01,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:01,375 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:07,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.45 % 2021-12-15 19:28:07,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:07,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:11,390 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:11,390 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.88 2021-12-15 19:28:11,407 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:11,410 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:11,411 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:17,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:17,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:17,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:21,421 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:21,421 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.88 2021-12-15 19:28:21,438 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:21,441 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:21,441 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:27,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:27,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:27,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:31,451 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:31,453 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.88 2021-12-15 19:28:31,469 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:31,471 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:31,471 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:37,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:37,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:37,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:41,482 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:41,482 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 52.88 2021-12-15 19:28:41,497 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:41,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:41,500 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:47,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.88 % 2021-12-15 19:28:47,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:47,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:51,509 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-15 19:28:51,510 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.35 2021-12-15 19:28:51,526 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:28:51,529 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:28:51,529 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:28:57,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-15 19:28:57,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:28:57,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:01,539 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-15 19:29:01,540 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.35 2021-12-15 19:29:01,558 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:01,562 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:01,562 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:07,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-15 19:29:07,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:07,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:11,572 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-15 19:29:11,572 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.35 2021-12-15 19:29:11,589 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:11,592 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:11,592 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:17,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:17,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:17,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:21,605 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:21,605 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.77 2021-12-15 19:29:21,621 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:21,624 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:21,624 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:27,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:27,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:27,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:31,634 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:31,634 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.77 2021-12-15 19:29:31,653 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:31,655 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:31,655 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:37,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:37,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:37,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:41,665 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:41,665 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 53.77 2021-12-15 19:29:41,684 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:41,687 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:41,687 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:47,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.77 % 2021-12-15 19:29:47,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:47,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:51,698 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:29:51,698 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:29:51,714 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:29:51,717 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:29:51,717 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:29:57,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:29:57,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:29:57,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:01,726 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:01,726 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:01,743 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:01,746 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:01,746 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:07,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:07,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:07,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:11,755 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:11,755 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:11,770 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:11,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:11,773 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:17,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:17,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:17,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:21,786 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:21,786 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:21,803 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:21,806 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:21,806 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:27,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:27,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:27,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:31,815 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:31,815 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:31,835 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:31,839 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:31,839 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:37,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:37,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:37,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:41,851 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:41,851 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:41,881 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:41,884 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:41,884 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:47,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:47,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:47,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:51,894 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.20 % 2021-12-15 19:30:51,894 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.2 2021-12-15 19:30:51,911 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:30:51,913 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:30:51,913 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:30:57,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.63 % 2021-12-15 19:30:57,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:30:57,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:01,923 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.63 % 2021-12-15 19:31:01,924 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.63 2021-12-15 19:31:01,940 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:01,941 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:01,941 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:07,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.63 % 2021-12-15 19:31:07,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:07,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:11,951 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.63 % 2021-12-15 19:31:11,951 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.63 2021-12-15 19:31:11,969 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:11,971 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:11,971 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:17,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:17,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:17,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:21,980 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:21,982 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.92 2021-12-15 19:31:21,997 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:22,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:22,000 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:27,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:27,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:27,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:32,009 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:32,009 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.92 2021-12-15 19:31:32,026 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:32,029 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:32,029 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:37,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:37,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:37,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:42,045 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.92 % 2021-12-15 19:31:42,046 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 54.92 2021-12-15 19:31:42,065 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:42,069 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:42,069 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:47,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:31:48,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:48,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:52,078 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:31:52,078 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.39 2021-12-15 19:31:52,094 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:31:52,095 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:31:52,095 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:31:58,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:31:58,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:31:58,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:02,105 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:32:02,105 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.39 2021-12-15 19:32:02,127 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:02,130 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:02,130 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:08,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:32:08,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:08,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:12,138 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.39 % 2021-12-15 19:32:12,140 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.39 2021-12-15 19:32:12,154 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:12,157 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:12,157 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:18,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:18,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:18,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:22,167 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:22,167 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.82 2021-12-15 19:32:22,183 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:22,187 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:22,187 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:28,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:28,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:28,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:32,197 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:32,197 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.82 2021-12-15 19:32:32,213 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:32,214 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:32,214 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:38,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:38,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:38,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:42,223 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:42,223 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.82 2021-12-15 19:32:42,239 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:42,240 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:42,240 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:48,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:48,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:48,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:52,252 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.82 % 2021-12-15 19:32:52,252 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 55.82 2021-12-15 19:32:52,266 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:32:52,267 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:32:52,269 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:32:58,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:32:58,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:32:58,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:02,276 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:02,278 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:02,292 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:02,293 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:02,293 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:08,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:08,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:08,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:12,302 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:12,303 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:12,319 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:12,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:12,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:18,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:18,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:18,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:22,332 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:22,332 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:22,349 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:22,351 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:22,351 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:28,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:28,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:28,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:32,362 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:32,364 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:32,381 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:32,384 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:32,384 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:38,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:38,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:38,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:42,395 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:42,395 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:42,414 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:42,417 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:42,417 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:48,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:48,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:48,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:52,430 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:52,430 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:33:52,450 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:33:52,453 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:33:52,453 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:33:58,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:33:58,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:33:58,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:02,464 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:34:02,464 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:34:02,483 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:02,484 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:02,486 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:08,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:34:08,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:08,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:12,499 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.24 % 2021-12-15 19:34:12,500 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.24 2021-12-15 19:34:12,516 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:12,519 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:12,519 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:18,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.67 % 2021-12-15 19:34:18,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:18,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:22,529 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.67 % 2021-12-15 19:34:22,529 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.67 2021-12-15 19:34:22,546 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:22,548 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:22,548 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:28,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.67 % 2021-12-15 19:34:28,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:28,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:32,558 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.67 % 2021-12-15 19:34:32,558 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.67 2021-12-15 19:34:32,575 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:32,579 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:32,579 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:38,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.96 % 2021-12-15 19:34:38,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:38,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:42,588 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.96 % 2021-12-15 19:34:42,589 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.96 2021-12-15 19:34:42,607 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:42,608 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:42,609 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:48,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.96 % 2021-12-15 19:34:48,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:48,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:52,618 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.96 % 2021-12-15 19:34:52,618 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 56.96 2021-12-15 19:34:52,635 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:34:52,638 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:34:52,640 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:34:58,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:34:58,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:34:58,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:02,651 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:02,651 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.43 2021-12-15 19:35:02,667 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:02,670 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:02,670 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:08,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:08,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:08,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:12,680 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:12,680 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.43 2021-12-15 19:35:12,694 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:12,696 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:12,696 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:18,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:18,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:18,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:22,706 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:22,706 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.43 2021-12-15 19:35:22,726 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:22,730 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:22,730 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:28,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:28,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:28,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:32,743 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.43 % 2021-12-15 19:35:32,743 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.43 2021-12-15 19:35:32,763 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:32,766 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:32,767 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:38,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.86 % 2021-12-15 19:35:38,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:38,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:42,778 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.86 % 2021-12-15 19:35:42,778 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.86 2021-12-15 19:35:42,795 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:42,796 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:42,796 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:48,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.86 % 2021-12-15 19:35:48,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:48,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:52,806 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.86 % 2021-12-15 19:35:52,808 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 57.86 2021-12-15 19:35:52,826 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:35:52,831 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:35:52,832 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:35:58,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.86 % 2021-12-15 19:35:58,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:35:58,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:02,842 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:02,844 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:02,861 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:02,864 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:02,864 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:08,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:08,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:08,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:12,872 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:12,874 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:12,888 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:12,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:12,891 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:18,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:18,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:18,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:22,904 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:22,904 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:22,923 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:22,926 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:22,926 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:28,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:28,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:28,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:32,937 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:32,937 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:32,953 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:32,957 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:32,957 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:38,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:38,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:38,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:42,967 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:42,967 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:42,983 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:42,984 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:42,986 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:48,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:48,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:48,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:52,994 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:52,996 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:36:53,012 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:36:53,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:36:53,015 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:36:58,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:36:58,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:36:58,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:03,026 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:37:03,026 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:37:03,043 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:03,046 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:03,046 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:08,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:37:08,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:08,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:13,056 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.29 % 2021-12-15 19:37:13,056 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.29 2021-12-15 19:37:13,072 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:13,075 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:13,075 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:18,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:18,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:18,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:23,088 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:23,088 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.71 2021-12-15 19:37:23,105 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:23,108 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:23,108 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:28,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:28,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:28,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:33,117 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:33,117 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.71 2021-12-15 19:37:33,134 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:33,137 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:33,138 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:38,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:38,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:38,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:43,148 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:43,148 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 58.71 2021-12-15 19:37:43,163 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:43,165 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:43,165 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:48,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.71 % 2021-12-15 19:37:48,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:48,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:53,174 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.18 % 2021-12-15 19:37:53,174 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.18 2021-12-15 19:37:53,190 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:37:53,193 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:37:53,193 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:37:58,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.18 % 2021-12-15 19:37:58,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:37:58,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:03,203 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:03,203 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.48 2021-12-15 19:38:03,221 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:03,224 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:03,226 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:08,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:08,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:08,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:13,236 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:13,236 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.48 2021-12-15 19:38:13,255 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:13,259 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:13,259 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:18,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:18,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:18,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:23,272 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:23,272 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.48 2021-12-15 19:38:23,290 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:23,293 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:23,293 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:28,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:28,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:28,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:33,302 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:33,302 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.48 2021-12-15 19:38:33,319 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:33,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:33,322 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:38,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.48 % 2021-12-15 19:38:38,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:38,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:43,332 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.90 % 2021-12-15 19:38:43,332 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.9 2021-12-15 19:38:43,348 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:43,349 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:43,349 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:48,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.90 % 2021-12-15 19:38:48,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:48,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:53,358 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.90 % 2021-12-15 19:38:53,358 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.9 2021-12-15 19:38:53,375 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:38:53,377 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:38:53,377 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:38:58,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.90 % 2021-12-15 19:38:58,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:38:58,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:39:03,387 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.90 % 2021-12-15 19:39:03,387 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 59.9 2021-12-15 19:39:03,403 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:39:03,404 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:39:03,405 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:39:08,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.33 % 2021-12-15 19:39:08,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:39:08,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:39:13,414 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.33 % 2021-12-15 19:39:13,414 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 60.33 2021-12-15 19:39:13,431 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:39:13,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:39:13,434 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:39:18,805 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >60.33% in 3356.4690001 seconds 2021-12-15 19:39:18,816 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 19:39:18,816 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 19:39:18,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 19:39:18,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:39:19,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 19:39:19,204 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:39:19,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 19:39:19,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 19:39:19,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 19:39:19,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 19:39:19,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 19:39:19,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 19:39:19,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 19:39:19,315 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 19:39:19,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 19:39:19,369 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['159417 memcached\n'], []) 2021-12-15 19:39:19,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 19:39:19,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 19:39:19,457 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['159417 memcached\n'], []) 2021-12-15 19:39:19,459 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 19:39:19,461 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 19:39:19,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:39:19,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 19:39:19,825 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:39:19,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 19:39:19,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 19:39:19,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 19:39:19,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 19:39:19,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 19:39:19,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 19:39:19,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 19:39:19,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:39:19,992 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 19:39:20,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:39:20,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:20,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:20,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:20,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:20,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:20,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:20,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:20,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:20,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:20,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:20,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:20,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:20,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:20,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:20,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:20,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:21,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:21,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:21,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:21,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:21,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:21,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:21,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:21,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:21,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:21,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:21,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:21,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:21,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:21,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:21,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:39:21,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:39:21,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:21,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:22,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:39:22,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:39:22,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:22,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:22,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 19:39:22,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 19:39:22,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:22,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:22,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 19:39:22,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 19:39:22,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:39:22,385 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 19:39:22,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:39:22,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:22,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:39:22,513 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 19:39:22,513 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 19:39:22,519 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 19:39:22,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:39:22,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 19:39:23,026 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:39:23,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 19:39:23,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 19:39:23,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 19:39:23,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 19:39:23,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 19:39:23,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 19:39:23,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 19:39:23,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:39:23,157 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 19:39:23,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:39:23,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:23,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:23,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:23,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:23,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,444 | test | DEBUG | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.33 % 2021-12-15 19:39:23,444 | test | INFO | pool-3-thread-27 | [task:check:385] Rebalance - status: running, progress: 60.33 2021-12-15 19:39:23,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,490 | infra | DEBUG | pool-3-thread-27 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:39:23,492 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:39:23,492 | infra | DEBUG | pool-3-thread-27 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:39:23,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:23,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:23,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:23,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:23,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:23,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:23,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:23,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:23,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:23,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:23,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:24,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:24,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:24,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:24,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:39:24,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:39:24,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:24,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:24,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:39:24,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:39:24,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:39:24,316 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 19:39:24,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:39:24,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:24,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:39:24,377 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 19:39:24,377 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 19:39:24,381 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 19:39:24,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:39:24,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 19:39:24,924 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:39:24,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 19:39:24,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 19:39:24,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 19:39:24,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 19:39:24,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 19:39:25,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 19:39:25,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 19:39:25,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:39:25,055 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 19:39:25,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:39:25,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:25,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:39:25,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:25,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:39:25,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:25,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:39:25,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:25,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:39:25,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:25,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:39:25,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:25,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:25,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:26,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:39:26,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:26,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:39:26,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:26,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:39:26,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:39:26,213 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 19:39:26,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:39:26,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:39:26,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:39:26,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 19:39:26,276 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 19:39:26,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 19:39:26,286 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 19:39:33,500 | test | ERROR | pool-3-thread-27 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0d4b41c413ba120beaaf755f27d87ce8', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d77e561a94252ad8d9d395ccddeb1025', u'status': u'notRunning'} - rebalance failed 2021-12-15 19:39:33,535 | test | INFO | pool-3-thread-27 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973244L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.244Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30976.151>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",967,\'ns_1@172.23.110.68\',\n [\'ns_1@172.23.110.65\',\'ns_1@172.23.110.64\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,451}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = b87a1daffb3d7d1d314234cf7fc2d051'} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973210L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.210Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 18:37:30.702273 159651 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973207L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.207Z', u'text': u'Worker <0.30832.151> (for action {move,\n {967,\n [\'ns_1@172.23.110.68\',\n \'ns_1@172.23.110.65\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.64\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.30976.151>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",\n 967,\n \'ns_1@172.23.110.68\',\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.64\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 451}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973179L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.179Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622482727L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:41:22.727Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622482570L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:41:22.570Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622482353L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:41:22.353Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = b87a1daffb3d7d1d314234cf7fc2d051"} 2021-12-15 19:39:33,536 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622454657L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:40:54.657Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 172 seconds.'} 2021-12-15 19:39:33,538 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246114L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.114Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 17:50:32.409045 155048 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 19:39:33,538 | test | ERROR | pool-3-thread-27 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639622246110L, u'shortText': u'message', u'serverTime': u'2021-12-15T18:37:26.110Z', u'text': u"Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {'EXIT',<0.31947.145>,\n {socket_closed,\n {gen_server,call,\n [{'janitor_agent-GleamBookUsers0',\n 'ns_1@172.23.110.64'},\n {if_rebalance,<0.23175.141>,\n {update_vbucket_state,441,active,\n undefined,undefined,\n [['ns_1@172.23.110.64',\n 'ns_1@172.23.110.68'],\n ['ns_1@172.23.110.64',\n 'ns_1@172.23.110.66']]}},\n infinity]}}}}}.\nRebalance Operation Id = cfedf5aeb5e515813e5c79c2e1ff50f9"} 2021-12-15 19:39:33,538 | test | ERROR | pool-3-thread-27 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'0d4b41c413ba120beaaf755f27d87ce8', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=d77e561a94252ad8d9d395ccddeb1025', u'status': u'notRunning'} - rebalance failed 2021-12-15 19:40:26,299 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 19:40:26,299 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 19:40:26,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:26,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:26,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:28,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:28,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:28,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:30,348 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:30,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:30,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:32,368 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:32,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:32,374 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:34,385 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:34,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:34,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:36,404 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:36,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:36,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:38,427 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:38,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:38,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:40,446 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:40,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:40,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:42,464 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:42,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:42,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:44,484 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:44,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:44,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:46,503 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:46,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:46,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:48,522 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:48,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:48,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:50,540 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:50,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:50,548 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:52,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:52,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:52,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:54,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:54,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:54,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:56,595 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:56,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:56,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:40:58,615 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:40:58,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:40:58,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:00,638 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:00,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:00,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:02,657 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:02,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:02,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:04,677 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:04,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:04,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:06,700 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:06,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:06,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:08,720 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:08,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:08,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:10,740 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:10,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:10,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:12,759 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:12,767 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:12,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:14,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:14,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:14,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:16,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:16,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:16,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:18,818 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:18,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:18,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:20,836 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:20,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:20,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:22,855 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:22,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:22,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:24,874 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:24,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:24,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:26,891 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:26,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:26,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:28,910 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:28,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:28,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:30,928 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:30,936 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:30,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:32,950 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:32,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:32,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:34,986 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:34,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:34,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:37,009 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:37,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:37,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:39,029 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:39,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:39,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:41,049 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:41,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:41,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:43,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:43,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:43,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:45,089 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:45,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:45,096 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:47,111 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:47,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:47,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:49,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:49,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:49,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:51,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:51,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:51,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:53,171 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:53,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:53,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:55,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:55,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:55,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:57,211 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:57,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:57,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:41:59,236 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:41:59,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:41:59,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:01,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:01,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:01,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:03,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:03,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:03,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:05,298 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:05,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:05,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:07,322 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:07,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:07,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:09,342 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:09,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:09,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:11,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:11,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:11,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:13,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:13,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:13,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:15,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:15,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:15,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:17,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:17,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:17,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:19,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:19,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:19,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:21,489 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:21,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:21,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:23,506 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:23,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:23,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:25,529 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:25,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:25,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:28,013 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:28,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:28,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:30,033 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:30,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:30,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:32,052 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:32,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:32,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:36,052 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:36,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:36,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:38,069 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:38,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:38,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:40,088 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:40,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:40,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:42,105 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:42,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:42,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:44,122 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:44,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:44,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:46,144 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:46,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:46,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:48,164 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:48,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:48,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:50,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:50,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:50,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:52,201 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:52,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:52,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:54,221 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:54,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:54,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:56,240 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:56,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:56,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:42:58,262 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:42:58,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:42:58,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:00,285 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:00,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:00,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:02,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:02,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:02,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:04,334 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:04,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:04,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:06,358 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:06,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:06,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:08,381 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:08,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:08,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:10,403 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:10,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:10,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:12,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:12,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:12,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:14,443 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:14,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:14,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:16,463 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:16,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:16,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:18,482 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:18,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:18,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:20,502 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:20,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:20,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:22,520 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:22,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:22,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:24,542 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:24,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:24,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:26,566 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:26,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:26,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:28,589 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:28,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:28,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:30,611 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:30,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:30,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:32,632 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:32,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:32,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:34,654 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:34,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:34,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:36,677 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:36,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:36,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:38,697 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:38,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:38,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:40,720 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:40,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:40,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:42,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:42,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:42,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:44,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 19:43:44,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 19:43:44,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 19:43:46,793 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 19:43:46,828 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 19:43:46,828 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 19:43:46,845 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 19:43:46,859 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 19:43:46,859 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 19:43:46,874 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 19:43:46,875 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 19:43:46,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:43:47,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 19:43:47,209 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:43:47,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 19:43:47,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 19:43:47,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 19:43:47,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 19:43:47,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 19:43:47,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 19:43:47,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 19:43:47,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:43:47,309 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 19:43:47,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:43:47,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:47,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:47,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:47,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:47,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:47,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:47,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:47,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:47,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:47,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:47,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:47,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:47,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:47,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:47,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:47,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:47,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:47,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:47,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:47,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:48,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:48,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:48,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:48,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:48,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:48,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:48,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:43:48,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:43:48,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 19:43:48,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 19:43:48,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 19:43:48,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 19:43:48,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:48,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 19:43:48,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 19:43:48,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:43:48,671 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 19:43:48,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:43:48,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:48,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:43:48,737 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 19:43:48,739 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 19:43:48,743 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 19:43:48,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:43:49,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 19:43:49,125 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:43:49,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 19:43:49,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 19:43:49,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 19:43:49,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 19:43:49,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 19:43:49,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 19:43:49,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 19:43:49,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:43:49,221 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 19:43:49,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:43:49,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:49,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:49,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:49,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:49,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:49,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:49,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:49,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:49,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:49,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:49,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:49,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:49,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:49,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:49,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:50,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:50,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:50,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:50,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:50,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:50,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:50,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:50,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:50,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:43:50,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 19:43:50,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:43:50,197 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 19:43:50,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:43:50,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:50,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:43:50,229 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 19:43:50,229 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 19:43:50,232 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 19:43:50,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 19:43:50,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 19:43:50,723 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 19:43:50,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 19:43:50,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 19:43:50,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 19:43:50,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 19:43:50,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 19:43:50,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 19:43:50,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 19:43:50,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 19:43:50,857 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 19:43:50,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 19:43:50,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:50,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:50,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:50,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 19:43:50,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:51,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 19:43:51,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:51,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 19:43:51,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:51,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 19:43:51,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:51,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 19:43:51,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:51,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 19:43:51,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:51,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 19:43:51,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 19:43:51,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:51,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 19:43:51,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 19:43:51,608 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 19:43:51,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 19:43:51,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 19:43:51,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 19:43:51,661 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 19:43:51,663 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 19:43:51,663 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639622482.29 2021-12-15 19:43:51,665 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 60 2021-12-15 19:43:51,694 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639626231.67 2021-12-15 19:43:51,696 | infra | INFO | pool-3-thread-28 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639626231.67' started 2021-12-15 19:43:51,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 19:43:51,707 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 19:43:51,734 | test | DEBUG | pool-3-thread-28 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-15 19:43:51,734 | test | DEBUG | pool-3-thread-28 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-15 19:43:51,792 | test | DEBUG | pool-3-thread-28 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 19:43:51,798 | test | INFO | pool-3-thread-28 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 3.28783514909 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.422468662275 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 2.81400041751 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 1.1941003934 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-15 19:43:51,806 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:43:51,806 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:43:51,829 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:43:51,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:43:51,834 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:01,861 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:01,862 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:01,878 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:01,881 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:01,881 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:11,891 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:11,891 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:11,908 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:11,911 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:11,911 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:21,921 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:21,921 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:21,937 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:21,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:21,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:31,956 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:31,956 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:31,973 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:31,976 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:31,976 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:41,986 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:41,986 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:42,002 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:42,005 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:42,005 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:44:52,015 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:44:52,015 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:44:52,032 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:44:52,036 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:44:52,036 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:02,046 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:45:02,046 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:45:02,062 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:02,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:02,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:12,075 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 19:45:12,075 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 19:45:12,094 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:12,098 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:12,098 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:22,108 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.51 % 2021-12-15 19:45:22,108 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 2.51 2021-12-15 19:45:22,124 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:22,127 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:22,127 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:32,135 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.51 % 2021-12-15 19:45:32,137 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 2.51 2021-12-15 19:45:32,151 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:32,155 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:32,155 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:42,165 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:45:42,165 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 3.07 2021-12-15 19:45:42,181 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:42,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:42,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:51,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:45:51,721 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 3.07 2021-12-15 19:45:51,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:45:51,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:45:51,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:45:52,194 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:45:52,194 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 3.07 2021-12-15 19:45:52,210 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:45:52,211 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:45:52,211 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:01,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:46:01,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:01,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:02,221 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.07 % 2021-12-15 19:46:02,221 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 3.07 2021-12-15 19:46:02,236 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:02,239 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:02,239 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:11,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:11,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:11,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:12,253 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:12,253 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:46:12,273 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:12,276 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:12,276 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:21,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:21,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:21,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:22,285 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:22,285 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:46:22,299 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:22,302 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:22,302 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:31,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:31,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:31,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:32,311 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:32,312 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:46:32,326 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:32,328 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:32,328 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:41,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:41,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:41,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:42,339 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:42,339 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:46:42,357 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:42,358 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:42,358 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:51,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:51,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:46:51,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:46:52,368 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:46:52,368 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:46:52,382 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:46:52,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:46:52,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:01,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:47:01,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:01,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:02,395 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.33 % 2021-12-15 19:47:02,395 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 4.33 2021-12-15 19:47:02,417 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:02,418 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:02,418 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:11,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.58 % 2021-12-15 19:47:11,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:11,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:12,427 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.58 % 2021-12-15 19:47:12,428 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 5.58 2021-12-15 19:47:12,446 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:12,447 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:12,447 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:21,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.58 % 2021-12-15 19:47:21,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:21,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:22,457 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.58 % 2021-12-15 19:47:22,459 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 5.58 2021-12-15 19:47:22,480 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:22,482 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:22,482 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:31,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.58 % 2021-12-15 19:47:31,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:31,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:32,492 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:47:32,493 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:47:32,510 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:32,513 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:32,513 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:41,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:47:41,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:41,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:42,523 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:47:42,523 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:47:42,542 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:42,543 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:42,543 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:52,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:47:52,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:47:52,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:47:52,553 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:47:52,553 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:47:52,572 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:47:52,573 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:47:52,573 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:02,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:02,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:02,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:02,582 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:02,582 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:48:02,599 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:02,601 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:02,601 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:12,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:12,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:12,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:12,611 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:12,611 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:48:12,641 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:12,642 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:12,642 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:22,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:22,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:22,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:22,654 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.87 % 2021-12-15 19:48:22,654 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 6.87 2021-12-15 19:48:22,674 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:22,676 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:22,676 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:32,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.13 % 2021-12-15 19:48:32,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:32,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:32,703 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.13 % 2021-12-15 19:48:32,703 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 8.13 2021-12-15 19:48:32,720 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:32,721 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:32,721 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:42,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 19:48:42,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:42,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:42,730 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.38 % 2021-12-15 19:48:42,732 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.38 2021-12-15 19:48:42,746 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:42,749 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:42,749 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:52,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:48:52,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:48:52,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:48:52,760 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:48:52,760 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.94 2021-12-15 19:48:52,779 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:48:52,782 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:48:52,782 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:02,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:02,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:02,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:02,792 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:02,792 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.94 2021-12-15 19:49:02,808 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:02,811 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:02,811 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:12,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:12,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:12,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:12,819 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:12,821 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.94 2021-12-15 19:49:12,839 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:12,842 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:12,844 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:22,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:22,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:22,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:22,854 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:22,854 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.94 2021-12-15 19:49:22,869 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:22,872 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:22,872 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:32,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:32,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:32,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:32,882 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.94 % 2021-12-15 19:49:32,884 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 9.94 2021-12-15 19:49:32,903 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:32,904 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:32,904 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:42,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:49:42,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:42,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:42,914 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:49:42,915 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:49:42,933 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:42,936 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:42,937 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:52,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:49:52,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:49:52,278 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:49:52,947 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:49:52,947 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:49:52,966 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:49:52,969 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:49:52,969 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:02,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:02,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:02,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:02,977 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:02,979 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:50:02,994 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:02,996 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:02,996 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:12,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:12,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:12,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:13,009 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:13,009 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:50:13,025 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:13,026 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:13,026 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:22,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:22,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:22,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:23,036 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:23,036 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:50:23,052 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:23,055 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:23,055 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:32,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:32,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:32,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:33,068 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.20 % 2021-12-15 19:50:33,069 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 11.2 2021-12-15 19:50:33,085 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:33,088 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:33,088 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:42,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:50:42,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:42,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:43,098 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:50:43,098 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 12.45 2021-12-15 19:50:43,128 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:43,131 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:43,131 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:52,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:50:52,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:50:52,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:50:53,140 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:50:53,140 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 12.45 2021-12-15 19:50:53,157 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:50:53,160 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:50:53,160 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:02,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:51:02,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:02,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:03,171 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:51:03,171 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 12.45 2021-12-15 19:51:03,194 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:03,198 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:03,200 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:12,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:51:12,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:12,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:13,210 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.45 % 2021-12-15 19:51:13,211 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 12.45 2021-12-15 19:51:13,230 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:13,234 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:13,234 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:22,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:22,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:22,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:23,243 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:23,244 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 13.01 2021-12-15 19:51:23,262 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:23,265 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:23,265 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:32,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:32,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:32,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:33,276 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:33,278 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 13.01 2021-12-15 19:51:33,295 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:33,298 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:33,298 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:42,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:42,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:42,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:43,308 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.01 % 2021-12-15 19:51:43,309 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 13.01 2021-12-15 19:51:43,323 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:43,325 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:43,326 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:52,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:51:52,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:51:52,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:51:53,338 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:51:53,338 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:51:53,357 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:51:53,359 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:51:53,359 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:02,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:02,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:02,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:03,371 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:03,371 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:03,387 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:03,388 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:03,388 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:12,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:12,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:12,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:13,398 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:13,398 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:13,413 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:13,414 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:13,415 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:22,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:22,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:22,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:23,426 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:23,426 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:23,443 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:23,444 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:23,444 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:32,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:32,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:32,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:33,467 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:33,467 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:33,483 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:33,484 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:33,484 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:42,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:42,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:42,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:43,494 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:43,494 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:43,515 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:43,516 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:43,516 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:52,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:52,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:52:52,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:52:53,528 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.27 % 2021-12-15 19:52:53,528 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 14.27 2021-12-15 19:52:53,549 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:52:53,551 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:52:53,552 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:02,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.52 % 2021-12-15 19:53:02,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:02,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:03,561 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.52 % 2021-12-15 19:53:03,561 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 15.52 2021-12-15 19:53:03,576 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:03,578 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:03,578 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:12,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.52 % 2021-12-15 19:53:12,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:12,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:13,588 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.52 % 2021-12-15 19:53:13,588 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 15.52 2021-12-15 19:53:13,605 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:13,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:13,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:22,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:22,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:22,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:23,617 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:23,618 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 19:53:23,634 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:23,637 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:23,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:32,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:32,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:32,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:33,648 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:33,648 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 19:53:33,665 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:33,667 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:33,667 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:42,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:42,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:42,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:43,677 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:43,677 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 19:53:43,694 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:43,696 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:43,696 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:52,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:52,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:53:52,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:53:53,704 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.81 % 2021-12-15 19:53:53,704 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 16.81 2021-12-15 19:53:53,720 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:53:53,721 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:53:53,723 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:02,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-15 19:54:02,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:02,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:03,733 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-15 19:54:03,733 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 17.36 2021-12-15 19:54:03,750 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:03,753 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:03,753 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:12,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-15 19:54:12,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:12,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:13,765 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-15 19:54:13,766 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 17.36 2021-12-15 19:54:13,790 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:13,792 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:13,793 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:22,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.62 % 2021-12-15 19:54:22,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:22,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:23,803 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.62 % 2021-12-15 19:54:23,803 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 18.62 2021-12-15 19:54:23,825 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:23,829 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:23,829 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:32,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.62 % 2021-12-15 19:54:32,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:32,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:33,841 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:54:33,842 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 19.88 2021-12-15 19:54:33,859 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:33,862 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:33,862 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:42,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:54:42,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:42,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:43,871 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:54:43,871 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 19.88 2021-12-15 19:54:43,888 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:43,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:43,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:52,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:54:52,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:54:52,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:54:53,901 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:54:53,901 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 19.88 2021-12-15 19:54:53,917 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:54:53,920 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:54:53,920 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:02,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:55:02,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:02,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:03,928 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:55:03,928 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 19.88 2021-12-15 19:55:03,944 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:03,947 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:03,947 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:12,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:55:12,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:12,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:13,960 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.88 % 2021-12-15 19:55:13,960 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 19.88 2021-12-15 19:55:13,977 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:13,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:13,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:22,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.16 % 2021-12-15 19:55:22,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:22,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:23,992 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.16 % 2021-12-15 19:55:23,992 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 21.16 2021-12-15 19:55:24,010 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:24,013 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:24,013 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:32,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.16 % 2021-12-15 19:55:32,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:32,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:34,025 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:55:34,025 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:55:34,042 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:34,043 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:34,045 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:42,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:55:42,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:42,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:44,055 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:55:44,056 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:55:44,075 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:44,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:44,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:52,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:55:52,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:55:52,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:55:54,091 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:55:54,091 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:55:54,109 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:55:54,112 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:55:54,112 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:02,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:02,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:02,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:04,124 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:04,124 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:56:04,141 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:04,144 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:04,144 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:12,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:12,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:12,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:14,155 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:14,157 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:56:14,176 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:14,178 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:14,178 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:22,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:22,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:22,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:24,188 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-15 19:56:24,190 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-15 19:56:24,207 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:24,210 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:24,211 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:32,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:32,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:32,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:34,220 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:34,221 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:56:34,237 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:34,240 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:34,240 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:42,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:42,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:42,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:44,250 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:44,250 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:56:44,266 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:44,269 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:44,269 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:52,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:53,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:56:53,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:56:54,279 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:56:54,279 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:56:54,296 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:56:54,298 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:56:54,298 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:03,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:03,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:03,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:04,308 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:04,308 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:57:04,325 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:04,328 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:04,328 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:13,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:13,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:13,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:14,338 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:14,338 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:57:14,354 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:14,357 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:14,357 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:23,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:23,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:23,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:24,367 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.23 % 2021-12-15 19:57:24,367 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 24.23 2021-12-15 19:57:24,381 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:24,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:24,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:33,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:33,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:33,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:34,398 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:34,398 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:57:34,415 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:34,418 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:34,418 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:43,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:43,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:43,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:44,428 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:44,428 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:57:44,444 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:44,446 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:44,446 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:53,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:53,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:57:53,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:57:54,457 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:57:54,457 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:57:54,476 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:57:54,480 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:57:54,480 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:03,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:03,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:03,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:04,490 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:04,490 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:58:04,506 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:04,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:04,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:13,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:13,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:13,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:14,516 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:14,516 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:58:14,533 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:14,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:14,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:23,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:23,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:23,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:24,545 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.49 % 2021-12-15 19:58:24,545 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 25.49 2021-12-15 19:58:24,559 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:24,561 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:24,561 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:33,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:33,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:33,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:34,569 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:34,571 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 26.75 2021-12-15 19:58:34,585 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:34,588 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:34,588 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:43,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:43,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:43,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:44,598 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:44,598 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 26.75 2021-12-15 19:58:44,617 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:44,619 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:44,619 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:53,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:53,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:58:53,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:58:54,630 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:58:54,631 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 26.75 2021-12-15 19:58:54,647 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:58:54,648 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:58:54,648 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:03,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:59:03,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:03,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:04,657 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.75 % 2021-12-15 19:59:04,657 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 26.75 2021-12-15 19:59:04,674 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:04,676 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:04,676 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:13,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:13,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:13,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:14,686 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:14,686 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 27.3 2021-12-15 19:59:14,706 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:14,707 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:14,707 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:23,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:23,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:23,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:24,720 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:24,720 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 27.3 2021-12-15 19:59:24,736 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:24,737 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:24,737 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:33,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:33,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:33,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:34,747 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.30 % 2021-12-15 19:59:34,747 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 27.3 2021-12-15 19:59:34,762 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:34,763 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:34,763 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:43,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 19:59:43,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:43,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:44,773 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 19:59:44,773 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 19:59:44,789 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:44,792 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:44,792 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:53,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 19:59:53,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 19:59:53,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 19:59:54,805 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 19:59:54,805 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 19:59:54,823 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 19:59:54,826 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 19:59:54,826 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:03,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:03,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:03,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:04,835 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:04,835 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 20:00:04,852 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:04,855 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:04,855 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:13,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:13,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:13,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:14,865 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:14,865 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 20:00:14,882 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:14,885 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:14,887 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:23,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:23,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:23,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:24,897 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:24,898 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 20:00:24,914 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:24,917 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:24,917 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:33,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:33,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:33,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:34,927 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:34,927 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 20:00:34,944 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:34,946 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:34,946 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:43,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:43,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:43,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:44,956 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.56 % 2021-12-15 20:00:44,956 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 28.56 2021-12-15 20:00:44,973 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:44,976 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:44,976 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:53,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:00:53,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:00:53,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:00:54,986 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:00:54,986 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 20:00:55,005 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:00:55,009 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:00:55,009 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:03,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:03,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:03,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:05,019 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:05,019 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 20:01:05,035 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:05,038 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:05,038 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:13,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:13,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:13,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:15,046 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:15,046 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 20:01:15,062 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:15,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:15,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:23,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:23,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:23,553 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:25,078 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:25,078 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 20:01:25,109 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:25,112 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:25,112 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:33,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:33,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:33,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:35,124 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-15 20:01:35,124 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-15 20:01:35,145 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:35,150 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:35,150 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:43,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.07 % 2021-12-15 20:01:43,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:43,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:45,163 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.07 % 2021-12-15 20:01:45,163 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.07 2021-12-15 20:01:45,180 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:45,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:45,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:53,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.07 % 2021-12-15 20:01:53,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:01:53,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:01:55,194 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.07 % 2021-12-15 20:01:55,194 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.07 2021-12-15 20:01:55,211 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:01:55,214 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:01:55,216 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:03,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.07 % 2021-12-15 20:02:03,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:03,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:05,224 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:05,224 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:05,242 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:05,244 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:05,244 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:13,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:13,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:13,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:15,256 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:15,256 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:15,272 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:15,275 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:15,275 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:23,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:23,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:23,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:25,283 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:25,285 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:25,301 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:25,303 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:25,303 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:33,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:33,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:33,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:35,315 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:35,315 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:35,332 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:35,334 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:35,335 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:43,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:43,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:43,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:45,344 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:45,344 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:45,359 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:45,362 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:45,362 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:53,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:53,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:02:53,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:02:55,374 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.63 % 2021-12-15 20:02:55,375 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 31.63 2021-12-15 20:02:55,388 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:02:55,391 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:02:55,391 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:03,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.89 % 2021-12-15 20:03:03,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:03,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:05,401 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.89 % 2021-12-15 20:03:05,401 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 32.89 2021-12-15 20:03:05,417 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:05,421 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:05,421 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:13,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:13,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:13,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:15,430 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:15,430 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:03:15,447 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:15,450 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:15,450 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:23,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:23,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:23,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:25,460 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:25,460 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:03:25,477 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:25,480 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:25,480 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:33,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:33,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:33,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:35,489 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:35,490 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:03:35,505 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:35,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:35,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:43,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:43,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:43,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:45,516 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:45,516 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:03:45,533 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:45,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:45,536 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:53,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:53,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:03:53,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:03:55,546 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:03:55,546 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:03:55,561 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:03:55,562 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:03:55,562 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:03,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:04:03,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:03,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:05,572 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:04:05,572 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:04:05,588 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:05,589 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:05,589 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:13,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:04:13,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:13,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:15,598 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.17 % 2021-12-15 20:04:15,598 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 34.17 2021-12-15 20:04:15,615 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:15,618 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:15,618 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:23,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.43 % 2021-12-15 20:04:23,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:23,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:25,628 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.43 % 2021-12-15 20:04:25,628 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 35.43 2021-12-15 20:04:25,642 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:25,644 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:25,644 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:33,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:33,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:33,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:35,654 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:35,654 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 37.97 2021-12-15 20:04:35,674 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:35,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:35,677 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:43,882 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:43,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:43,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:45,687 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:45,687 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 37.97 2021-12-15 20:04:45,703 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:45,704 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:45,704 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:53,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:53,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:04:53,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:04:55,716 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:04:55,717 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 37.97 2021-12-15 20:04:55,733 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:04:55,734 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:04:55,736 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:03,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:05:03,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:03,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:05,749 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:05:05,749 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 37.97 2021-12-15 20:05:05,766 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:05,767 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:05,767 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:13,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:05:13,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:13,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:15,779 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:05:15,779 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 37.97 2021-12-15 20:05:15,802 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:15,803 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:15,803 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:23,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.97 % 2021-12-15 20:05:23,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:23,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:25,816 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:25,816 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:05:25,832 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:25,835 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:25,835 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:33,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:33,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:33,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:35,846 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:35,848 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:05:35,878 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:35,880 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:35,880 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:43,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:43,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:43,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:45,891 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:45,891 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:05:45,911 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:45,914 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:45,914 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:54,005 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:54,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:05:54,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:05:55,924 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:05:55,924 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:05:55,941 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:05:55,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:05:55,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:04,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:04,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:04,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:05,953 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:05,953 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:06:05,970 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:05,971 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:05,971 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:14,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:14,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:14,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:15,982 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:15,982 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:06:15,997 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:16,000 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:16,000 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:24,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:24,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:24,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:26,009 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.23 % 2021-12-15 20:06:26,009 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 39.23 2021-12-15 20:06:26,023 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:26,026 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:26,026 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:34,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:34,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:34,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:36,036 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:36,038 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:06:36,053 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:36,058 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:36,058 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:44,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:44,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:44,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:46,066 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:46,068 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:06:46,084 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:46,086 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:46,086 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:54,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:54,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:06:54,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:06:56,117 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:06:56,117 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:06:56,134 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:06:56,137 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:06:56,138 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:04,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:04,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:04,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:06,147 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:06,147 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:07:06,163 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:06,165 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:06,165 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:14,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:14,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:14,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:16,174 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:16,176 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:07:16,194 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:16,196 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:16,196 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:24,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:24,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:24,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:26,207 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:26,207 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:07:26,223 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:26,226 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:26,226 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:34,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:34,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:34,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:36,236 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.49 % 2021-12-15 20:07:36,236 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 40.49 2021-12-15 20:07:36,253 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:36,256 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:36,256 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:44,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.03 % 2021-12-15 20:07:44,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:44,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:46,265 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.03 % 2021-12-15 20:07:46,266 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 43.03 2021-12-15 20:07:46,282 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:46,285 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:46,285 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:54,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.03 % 2021-12-15 20:07:54,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:07:54,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:07:56,296 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.03 % 2021-12-15 20:07:56,298 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 43.03 2021-12-15 20:07:56,315 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:07:56,318 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:07:56,318 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:04,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-15 20:08:04,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:04,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:06,328 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-15 20:08:06,328 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.32 2021-12-15 20:08:06,345 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:06,348 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:06,348 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:14,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-15 20:08:14,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:14,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:16,358 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-15 20:08:16,358 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.32 2021-12-15 20:08:16,375 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:16,378 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:16,378 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:24,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-15 20:08:24,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:24,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:26,387 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:26,388 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.87 2021-12-15 20:08:26,404 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:26,405 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:26,405 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:34,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:34,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:34,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:36,418 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:36,418 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.87 2021-12-15 20:08:36,437 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:36,441 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:36,441 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:44,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:44,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:44,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:46,460 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:46,460 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.87 2021-12-15 20:08:46,474 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:46,477 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:46,477 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:54,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:54,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:08:54,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:08:56,487 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:08:56,487 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.87 2021-12-15 20:08:56,505 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:08:56,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:08:56,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:04,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:09:04,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:04,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:06,519 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.87 % 2021-12-15 20:09:06,519 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 44.87 2021-12-15 20:09:06,535 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:06,538 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:06,538 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:14,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.13 % 2021-12-15 20:09:14,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:14,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:16,549 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.13 % 2021-12-15 20:09:16,549 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 46.13 2021-12-15 20:09:16,568 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:16,571 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:16,571 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:24,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:24,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:24,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:26,582 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:26,582 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 47.39 2021-12-15 20:09:26,599 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:26,601 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:26,601 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:34,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:34,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:34,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:36,611 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:36,611 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 47.39 2021-12-15 20:09:36,628 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:36,630 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:36,630 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:44,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:44,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:44,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:46,638 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:46,640 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 47.39 2021-12-15 20:09:46,654 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:46,657 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:46,657 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:54,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:54,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:09:54,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:09:56,665 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:09:56,665 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 47.39 2021-12-15 20:09:56,680 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:09:56,683 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:09:56,683 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:04,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:10:04,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:04,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:06,691 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.39 % 2021-12-15 20:10:06,691 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 47.39 2021-12-15 20:10:06,710 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:06,713 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:06,713 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:14,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:14,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:14,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:16,724 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:16,724 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 48.64 2021-12-15 20:10:16,740 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:16,742 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:16,742 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:24,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:24,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:24,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:26,753 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:26,753 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 48.64 2021-12-15 20:10:26,770 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:26,773 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:26,773 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:34,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:34,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:34,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:36,782 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.64 % 2021-12-15 20:10:36,782 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 48.64 2021-12-15 20:10:36,799 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:36,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:36,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:44,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:10:44,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:44,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:46,811 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:10:46,811 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 49.93 2021-12-15 20:10:46,829 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:46,831 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:46,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:54,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:10:54,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:10:54,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:10:56,842 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:10:56,842 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 49.93 2021-12-15 20:10:56,858 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:10:56,859 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:10:56,861 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:04,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:04,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:04,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:06,885 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:06,887 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 49.93 2021-12-15 20:11:06,905 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:06,907 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:06,907 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:14,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:14,617 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:14,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:16,918 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:16,918 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 49.93 2021-12-15 20:11:16,937 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:16,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:16,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:24,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:24,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:24,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:26,950 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:26,951 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 49.93 2021-12-15 20:11:26,970 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:26,973 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:26,973 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:34,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.93 % 2021-12-15 20:11:34,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:34,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:36,982 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:11:36,982 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 51.19 2021-12-15 20:11:37,002 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:37,005 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:37,005 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:44,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:11:44,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:44,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:47,015 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:11:47,015 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 51.19 2021-12-15 20:11:47,033 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:47,036 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:47,038 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:54,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:11:54,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:11:54,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:11:57,046 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:11:57,046 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 51.19 2021-12-15 20:11:57,062 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:11:57,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:11:57,065 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:04,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.19 % 2021-12-15 20:12:04,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:04,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:07,075 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.44 % 2021-12-15 20:12:07,076 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 52.44 2021-12-15 20:12:07,092 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:07,095 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:07,095 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:14,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.44 % 2021-12-15 20:12:14,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:14,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:17,105 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.44 % 2021-12-15 20:12:17,105 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 52.44 2021-12-15 20:12:17,121 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:17,125 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:17,125 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:24,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.44 % 2021-12-15 20:12:24,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:24,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:27,135 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.44 % 2021-12-15 20:12:27,135 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 52.44 2021-12-15 20:12:27,153 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:27,154 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:27,154 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:34,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:34,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:34,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:37,164 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:37,164 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 53.73 2021-12-15 20:12:37,180 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:37,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:37,184 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:44,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:44,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:44,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:47,194 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:47,194 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 53.73 2021-12-15 20:12:47,210 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:47,211 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:47,213 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:54,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:54,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:12:54,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:12:57,221 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.73 % 2021-12-15 20:12:57,221 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 53.73 2021-12-15 20:12:57,237 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:12:57,240 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:12:57,240 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:04,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:04,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:04,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:07,250 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:07,250 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 54.28 2021-12-15 20:13:07,266 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:07,267 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:07,267 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:14,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:14,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:14,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:17,279 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:17,279 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 54.28 2021-12-15 20:13:17,293 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:17,296 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:17,296 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:24,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:24,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:24,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:27,305 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:27,306 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 54.28 2021-12-15 20:13:27,322 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:27,325 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:27,325 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:34,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:34,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:34,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:37,335 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:37,335 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 54.28 2021-12-15 20:13:37,351 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:37,354 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:37,354 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:44,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-15 20:13:44,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:44,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:47,362 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:13:47,364 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-15 20:13:47,380 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:47,382 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:47,382 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:54,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:13:54,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:13:54,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:13:57,404 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:13:57,404 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-15 20:13:57,421 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:13:57,424 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:13:57,424 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:04,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:14:04,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:04,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:07,434 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:14:07,434 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-15 20:14:07,448 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:07,451 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:07,451 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:14,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:14:14,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:14,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:17,460 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-15 20:14:17,460 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-15 20:14:17,476 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:17,479 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:17,479 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:24,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.86 % 2021-12-15 20:14:24,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:24,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:27,489 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.86 % 2021-12-15 20:14:27,489 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 56.86 2021-12-15 20:14:27,505 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:27,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:27,507 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:34,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.86 % 2021-12-15 20:14:34,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:34,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:37,519 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.86 % 2021-12-15 20:14:37,519 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 56.86 2021-12-15 20:14:37,539 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:37,542 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:37,542 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:44,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:14:44,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:44,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:47,552 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:14:47,552 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.41 2021-12-15 20:14:47,575 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:47,578 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:47,578 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:55,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:14:55,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:14:55,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:14:57,588 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:14:57,589 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.41 2021-12-15 20:14:57,605 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:14:57,607 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:14:57,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:05,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:15:05,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:05,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:07,617 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:15:07,617 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.41 2021-12-15 20:15:07,631 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:07,634 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:07,634 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:15,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:15:15,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:15,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:17,642 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:15:17,642 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.41 2021-12-15 20:15:17,657 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:17,661 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:17,661 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:25,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-15 20:15:25,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:25,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:27,668 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.97 % 2021-12-15 20:15:27,670 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.97 2021-12-15 20:15:27,683 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:27,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:27,684 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:35,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.97 % 2021-12-15 20:15:35,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:35,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:37,694 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.97 % 2021-12-15 20:15:37,694 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.97 2021-12-15 20:15:37,709 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:37,710 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:37,710 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:45,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.97 % 2021-12-15 20:15:45,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:45,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:47,720 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.97 % 2021-12-15 20:15:47,720 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 57.97 2021-12-15 20:15:47,734 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:47,736 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:47,736 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:55,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-15 20:15:55,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:15:55,137 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:15:57,746 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-15 20:15:57,746 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-15 20:15:57,763 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:15:57,765 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:15:57,765 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:05,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:05,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:05,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:07,773 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:07,773 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 60.54 2021-12-15 20:16:07,789 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:07,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:07,790 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:15,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:15,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:15,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:17,801 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:17,801 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 60.54 2021-12-15 20:16:17,825 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:17,826 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:17,826 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:25,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:25,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:25,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:27,838 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:27,838 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 60.54 2021-12-15 20:16:27,857 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:27,858 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:27,858 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:35,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:35,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:35,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:37,869 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:37,871 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 60.54 2021-12-15 20:16:37,890 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:37,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:37,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:45,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:45,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:45,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:47,901 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:47,903 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 60.54 2021-12-15 20:16:47,924 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:47,926 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:47,926 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:55,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.54 % 2021-12-15 20:16:55,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:16:55,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:16:57,937 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:16:57,937 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 61.09 2021-12-15 20:16:57,956 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:16:57,957 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:16:57,957 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:05,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:05,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:05,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:07,967 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:07,967 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 61.09 2021-12-15 20:17:07,986 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:07,989 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:07,989 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:15,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:15,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:15,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:17,999 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:18,000 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 61.09 2021-12-15 20:17:18,019 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:18,020 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:18,022 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:25,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:25,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:25,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:28,032 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:28,032 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 61.09 2021-12-15 20:17:28,051 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:28,052 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:28,053 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:35,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:35,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:35,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:38,065 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:38,065 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 61.09 2021-12-15 20:17:38,084 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:38,085 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:38,085 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:45,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.09 % 2021-12-15 20:17:45,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:45,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:48,095 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.38 % 2021-12-15 20:17:48,095 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 62.38 2021-12-15 20:17:48,111 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:48,111 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:48,112 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:55,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.67 % 2021-12-15 20:17:55,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:17:55,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:17:58,121 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.67 % 2021-12-15 20:17:58,121 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 63.67 2021-12-15 20:17:58,137 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:17:58,140 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:17:58,140 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:05,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:05,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:05,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:08,160 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:08,160 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:08,176 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:08,177 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:08,177 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:15,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:15,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:15,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:18,187 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:18,187 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:18,203 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:18,204 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:18,206 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:25,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:25,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:25,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:28,213 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:28,214 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:28,230 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:28,233 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:28,233 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:35,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:35,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:35,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:38,253 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:38,255 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:38,270 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:38,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:38,273 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:45,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:45,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:45,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:48,290 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:48,292 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:48,306 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:48,309 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:48,309 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:55,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:55,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:18:55,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:18:58,319 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:18:58,319 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:18:58,335 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:18:58,338 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:18:58,338 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:05,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:05,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:05,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:08,348 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:08,348 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:19:08,364 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:08,367 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:08,367 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:15,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:15,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:15,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:18,375 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:18,377 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:19:18,395 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:18,400 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:18,400 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:25,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:25,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:25,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:28,410 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.22 % 2021-12-15 20:19:28,410 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 64.22 2021-12-15 20:19:28,424 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:28,427 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:28,427 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:35,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.51 % 2021-12-15 20:19:35,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:35,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:38,437 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.51 % 2021-12-15 20:19:38,437 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 65.51 2021-12-15 20:19:38,451 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:38,454 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:38,456 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:45,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.51 % 2021-12-15 20:19:45,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:45,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:48,464 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:19:48,464 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 66.79 2021-12-15 20:19:48,480 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:48,483 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:48,483 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:55,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:19:55,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:19:55,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:19:58,493 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:19:58,493 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 66.79 2021-12-15 20:19:58,512 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:19:58,513 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:19:58,515 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:05,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:20:05,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:05,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:08,526 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:20:08,526 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 66.79 2021-12-15 20:20:08,542 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:08,546 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:08,546 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:15,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:20:15,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:15,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:18,558 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:20:18,559 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 66.79 2021-12-15 20:20:18,575 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:18,578 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:18,579 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:25,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.79 % 2021-12-15 20:20:25,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:25,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:28,592 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:28,592 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:20:28,608 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:28,611 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:28,611 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:35,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:35,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:35,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:38,619 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:38,621 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:20:38,637 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:38,640 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:38,640 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:45,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:45,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:45,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:48,651 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:48,651 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:20:48,667 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:48,671 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:48,671 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:55,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:55,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:20:55,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:20:58,680 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:20:58,680 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:20:58,697 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:20:58,698 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:20:58,700 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:05,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:05,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:05,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:08,707 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:08,709 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:08,723 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:08,726 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:08,726 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:15,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:15,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:15,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:18,734 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:18,734 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:18,749 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:18,750 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:18,750 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:25,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:25,775 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:25,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:28,759 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:28,760 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:28,792 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:28,793 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:28,795 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:35,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:35,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:35,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:38,805 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:38,806 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:38,821 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:38,822 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:38,822 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:45,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:45,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:45,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:48,834 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:48,835 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:48,867 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:48,869 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:48,869 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:55,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:55,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:21:55,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:21:58,881 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:21:58,881 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:21:58,904 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:21:58,905 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:21:58,905 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:05,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:05,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:05,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:08,917 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:08,917 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:22:08,938 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:08,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:08,940 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:15,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:15,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:15,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:18,950 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:18,950 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:22:18,966 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:18,967 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:18,969 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:25,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:25,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:25,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:28,979 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:28,979 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:22:29,000 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:29,003 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:29,003 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:35,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:35,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:35,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:39,013 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.35 % 2021-12-15 20:22:39,013 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.35 2021-12-15 20:22:39,033 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:39,036 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:39,036 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:45,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:22:45,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:45,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:49,049 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:22:49,049 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.9 2021-12-15 20:22:49,069 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:49,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:49,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:55,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:22:55,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:22:55,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:22:59,082 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:22:59,082 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.9 2021-12-15 20:22:59,098 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:22:59,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:22:59,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:05,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:23:05,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:05,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:09,111 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:23:09,111 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.9 2021-12-15 20:23:09,131 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:09,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:09,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:15,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:23:15,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:15,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:19,142 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.90 % 2021-12-15 20:23:19,142 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 67.9 2021-12-15 20:23:19,168 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:19,171 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:19,171 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:25,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.19 % 2021-12-15 20:23:25,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:25,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:29,181 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.19 % 2021-12-15 20:23:29,181 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 69.19 2021-12-15 20:23:29,197 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:29,200 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:29,200 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:35,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.19 % 2021-12-15 20:23:36,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:36,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:39,209 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.19 % 2021-12-15 20:23:39,209 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 69.19 2021-12-15 20:23:39,224 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:39,226 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:39,226 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:46,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.19 % 2021-12-15 20:23:46,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:46,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:49,236 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:23:49,236 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:23:49,253 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:49,256 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:49,256 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:56,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:23:56,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:23:56,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:23:59,266 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:23:59,266 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:23:59,282 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:23:59,285 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:23:59,285 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:06,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:06,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:06,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:09,293 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:09,295 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:09,309 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:09,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:09,312 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:16,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:16,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:16,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:19,321 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:19,322 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:19,336 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:19,338 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:19,338 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:26,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:26,096 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:26,096 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:29,348 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:29,349 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:29,367 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:29,368 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:29,368 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:36,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:36,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:36,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:39,378 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:39,378 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:39,395 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:39,398 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:39,398 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:46,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:46,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:46,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:49,407 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:49,407 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:49,426 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:49,428 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:49,428 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:56,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:56,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:24:56,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:24:59,440 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.48 % 2021-12-15 20:24:59,440 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 70.48 2021-12-15 20:24:59,457 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:24:59,460 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:24:59,460 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:06,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:06,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:06,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:09,470 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:09,470 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:09,486 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:09,490 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:09,490 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:16,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:16,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:16,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:19,500 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:19,500 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:19,516 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:19,519 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:19,519 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:26,193 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:26,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:26,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:29,529 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:29,529 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:29,546 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:29,549 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:29,549 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:36,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:36,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:36,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:39,559 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:39,559 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:39,578 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:39,582 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:39,582 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:46,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:46,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:46,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:49,591 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:49,592 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:49,607 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:49,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:49,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:56,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:56,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:25:56,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:25:59,618 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:25:59,619 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:25:59,634 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:25:59,637 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:25:59,637 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:06,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:06,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:06,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:09,645 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:09,647 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:09,663 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:09,664 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:09,665 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:16,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:16,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:16,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:19,674 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:19,674 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:19,688 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:19,691 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:19,691 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:26,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:26,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:26,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:29,701 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:29,703 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:29,717 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:29,720 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:29,720 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:36,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:36,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:36,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:39,730 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:39,730 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:39,744 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:39,746 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:39,747 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:46,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:46,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:46,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:49,756 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:49,756 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:49,775 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:49,778 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:49,778 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:56,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:56,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:26:56,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:26:59,786 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:26:59,786 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:26:59,802 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:26:59,805 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:26:59,805 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:06,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:06,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:06,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:09,815 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:09,815 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:27:09,831 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:09,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:09,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:16,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:16,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:16,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:19,842 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:19,844 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:27:19,859 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:19,861 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:19,861 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:26,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:26,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:26,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:29,871 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:29,871 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:27:29,890 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:29,892 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:29,892 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:36,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:36,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:36,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:39,904 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:39,904 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:27:39,921 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:39,923 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:39,923 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:46,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:46,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:46,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:49,934 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:49,934 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.03 2021-12-15 20:27:49,948 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:49,950 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:49,950 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:56,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.03 % 2021-12-15 20:27:56,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:27:56,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:27:59,960 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:27:59,960 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.59 2021-12-15 20:27:59,973 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:27:59,974 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:27:59,974 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:06,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:06,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:06,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:09,984 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:09,984 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.59 2021-12-15 20:28:09,999 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:28:10,000 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:28:10,000 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:16,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:16,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:16,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:20,010 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:20,012 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.59 2021-12-15 20:28:20,029 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:28:20,032 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:28:20,032 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:26,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:26,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:26,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:30,040 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:30,042 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.59 2021-12-15 20:28:30,056 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:28:30,059 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:28:30,059 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:36,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:36,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:36,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:40,082 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.59 % 2021-12-15 20:28:40,082 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 71.59 2021-12-15 20:28:40,098 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:28:40,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:28:40,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:46,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 20:28:46,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:46,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:50,114 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 20:28:50,115 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 72.87 2021-12-15 20:28:50,131 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:28:50,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:28:50,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:28:56,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 20:28:56,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:28:56,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:00,144 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 20:29:00,144 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 72.87 2021-12-15 20:29:00,158 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:00,161 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:00,161 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:06,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.87 % 2021-12-15 20:29:06,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:06,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:10,171 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:10,171 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:29:10,187 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:10,188 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:10,188 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:16,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:16,665 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:16,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:20,200 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:20,200 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:29:20,219 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:20,220 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:20,220 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:26,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:26,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:26,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:30,232 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:30,233 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:29:30,247 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:30,250 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:30,250 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:36,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:36,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:36,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:40,263 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:40,263 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:29:40,280 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:40,282 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:40,283 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:46,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:46,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:46,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:50,295 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:50,296 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:29:50,315 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:29:50,319 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:29:50,319 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:29:56,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:29:56,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:29:56,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:00,332 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:30:00,332 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:30:00,349 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:00,351 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:00,352 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:06,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:30:06,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:06,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:10,365 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:30:10,365 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:30:10,381 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:10,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:10,384 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:16,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:30:16,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:16,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:20,397 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.16 % 2021-12-15 20:30:20,398 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.16 2021-12-15 20:30:20,414 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:20,417 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:20,417 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:26,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:26,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:26,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:30,428 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:30,430 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:30:30,444 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:30,447 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:30,447 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:36,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:36,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:36,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:40,457 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:40,457 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:30:40,473 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:40,476 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:40,476 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:46,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:46,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:46,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:50,489 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:50,489 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:30:50,515 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:30:50,517 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:30:50,517 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:30:56,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:30:56,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:30:56,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:00,529 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:00,529 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:00,546 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:00,549 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:00,549 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:06,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:06,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:06,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:10,561 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:10,562 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:10,578 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:10,579 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:10,581 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:16,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:16,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:16,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:20,591 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:20,591 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:20,607 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:20,608 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:20,609 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:26,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:26,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:26,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:30,619 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:30,619 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:30,634 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:30,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:30,638 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:36,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:36,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:36,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:40,648 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:40,648 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:40,667 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:40,670 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:40,670 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:46,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:46,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:46,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:50,684 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:50,684 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:31:50,703 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:31:50,711 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:31:50,713 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:31:56,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:31:56,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:31:56,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:00,723 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:00,723 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:00,740 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:00,743 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:00,743 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:06,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:06,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:06,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:10,755 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:10,755 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:10,770 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:10,773 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:10,773 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:17,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:17,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:17,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:20,783 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:20,783 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:20,799 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:20,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:20,801 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:27,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:27,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:27,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:30,813 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:30,813 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:30,828 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:30,829 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:30,829 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:37,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:37,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:37,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:40,839 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:40,841 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:40,855 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:40,857 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:40,857 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:47,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:47,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:47,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:50,867 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:50,867 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:32:50,882 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:32:50,884 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:32:50,885 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:32:57,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:32:57,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:32:57,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:00,897 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:00,897 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:33:00,911 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:00,914 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:00,914 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:07,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:07,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:07,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:10,926 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:10,927 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:33:10,941 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:10,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:10,944 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:17,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:17,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:17,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:20,954 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:20,956 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 74.71 2021-12-15 20:33:20,976 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:20,979 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:20,979 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:27,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.71 % 2021-12-15 20:33:27,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:27,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:30,990 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:30,990 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 75.27 2021-12-15 20:33:31,013 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:31,015 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:31,015 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:37,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:37,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:37,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:41,026 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:41,026 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 75.27 2021-12-15 20:33:41,046 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:41,048 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:41,048 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:47,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:47,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:47,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:51,058 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:51,058 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 75.27 2021-12-15 20:33:51,075 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:33:51,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:33:51,078 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:33:57,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:33:57,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:33:57,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:01,088 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:34:01,088 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 75.27 2021-12-15 20:34:01,105 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:01,108 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:01,108 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:07,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.27 % 2021-12-15 20:34:07,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:07,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:11,117 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:11,118 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 76.55 2021-12-15 20:34:11,134 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:11,135 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:11,135 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:17,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:17,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:17,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:21,148 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:21,148 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 76.55 2021-12-15 20:34:21,161 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:21,163 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:21,163 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:27,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:27,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:27,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:31,174 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:31,174 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 76.55 2021-12-15 20:34:31,190 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:31,191 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:31,193 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:37,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:37,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:37,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:41,201 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-15 20:34:41,201 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 76.55 2021-12-15 20:34:41,217 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:41,219 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:41,219 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:47,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:34:47,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:47,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:51,229 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:34:51,230 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:34:51,250 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:34:51,252 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:34:51,252 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:34:57,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:34:57,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:34:57,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:01,263 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:01,263 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:01,283 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:01,286 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:01,286 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:07,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:07,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:07,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:11,299 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:11,299 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:11,321 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:11,323 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:11,323 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:17,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:17,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:17,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:21,335 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:21,335 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:21,351 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:21,355 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:21,355 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:27,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:27,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:27,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:31,365 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:31,367 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:31,382 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:31,385 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:31,385 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:37,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:37,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:37,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:41,395 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:41,397 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:41,411 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:41,415 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:41,415 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:47,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:47,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:47,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:51,428 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:51,430 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 77.84 2021-12-15 20:35:51,447 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:35:51,448 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:35:51,448 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:35:57,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.84 % 2021-12-15 20:35:57,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:35:57,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:01,460 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:01,460 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:01,476 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:01,477 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:01,477 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:07,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:07,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:07,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:11,489 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:11,489 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:11,506 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:11,509 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:11,509 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:17,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:17,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:17,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:21,520 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:21,520 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:21,536 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:21,539 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:21,539 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:27,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:27,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:27,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:31,549 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:31,549 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:31,565 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:31,568 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:31,568 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:37,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:37,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:37,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:41,578 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:41,578 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:41,594 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:41,596 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:41,596 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:47,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:47,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:47,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:51,608 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:51,608 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:36:51,622 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:36:51,625 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:36:51,625 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:36:57,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:36:57,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:36:57,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:01,635 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:01,635 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:01,651 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:01,654 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:01,654 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:07,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:07,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:07,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:11,664 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:11,664 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:11,680 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:11,683 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:11,683 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:17,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:17,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:17,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:21,709 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:21,709 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:21,729 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:21,733 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:21,733 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:27,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:27,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:27,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:31,746 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:31,746 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:31,765 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:31,769 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:31,769 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:37,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:37,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:37,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:41,779 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:41,780 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:41,796 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:41,799 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:41,799 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:47,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:47,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:47,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:51,809 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:51,809 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:37:51,828 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:37:51,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:37:51,832 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:37:57,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:37:57,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:37:57,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:01,842 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:01,842 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:01,857 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:01,858 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:01,859 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:07,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:07,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:07,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:11,871 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:11,871 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:11,888 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:11,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:11,891 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:17,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:17,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:17,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:21,901 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:21,903 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:21,920 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:21,921 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:21,923 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:27,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:27,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:27,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:31,933 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:31,933 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:31,948 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:31,950 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:31,950 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:37,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:37,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:37,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:41,961 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:41,961 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:41,979 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:41,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:41,980 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:47,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:47,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:47,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:51,992 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:51,992 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.4 2021-12-15 20:38:52,006 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:38:52,009 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:38:52,009 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:38:57,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.40 % 2021-12-15 20:38:57,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:38:57,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:02,022 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:02,023 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.95 2021-12-15 20:39:02,038 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:02,040 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:02,040 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:07,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:07,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:39:07,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:12,051 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:12,051 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.95 2021-12-15 20:39:12,069 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:12,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:12,072 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:17,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:17,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:39:17,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:22,082 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:22,082 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 78.95 2021-12-15 20:39:22,098 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:22,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:22,101 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:27,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.95 % 2021-12-15 20:39:27,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:39:27,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:32,112 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.24 % 2021-12-15 20:39:32,112 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 80.24 2021-12-15 20:39:32,131 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:32,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:32,132 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:37,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.24 % 2021-12-15 20:39:37,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 20:39:37,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:42,144 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.24 % 2021-12-15 20:39:42,144 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 80.24 2021-12-15 20:39:42,158 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:42,161 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:42,161 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:47,845 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >80.24% in 3236.12300014 seconds 2021-12-15 20:39:47,855 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 20:39:47,857 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 20:39:47,858 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 20:39:47,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:39:48,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 20:39:48,194 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:39:48,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 20:39:48,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 20:39:48,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 20:39:48,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 20:39:48,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 20:39:48,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 20:39:48,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 20:39:48,288 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 20:39:48,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 20:39:48,349 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['165016 memcached\n'], []) 2021-12-15 20:39:48,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 20:39:48,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 20:39:48,467 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['165016 memcached\n'], []) 2021-12-15 20:39:48,469 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 20:39:48,470 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 20:39:48,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:39:48,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 20:39:48,828 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:39:48,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 20:39:48,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 20:39:48,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 20:39:48,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 20:39:48,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 20:39:48,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 20:39:48,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 20:39:48,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:39:48,957 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 20:39:48,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:39:49,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:49,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:49,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:49,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:49,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:49,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:49,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:49,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:49,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:49,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:49,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:49,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:49,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:49,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:49,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:49,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:49,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:49,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:49,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:39:49,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:39:49,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:50,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:50,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:39:50,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:39:50,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:50,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:50,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 20:39:50,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 20:39:50,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:50,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:50,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 20:39:50,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 20:39:50,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:39:50,233 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 20:39:50,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:39:50,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:50,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:39:50,335 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 20:39:50,335 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 20:39:50,338 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 20:39:50,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:39:50,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 20:39:50,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:39:50,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 20:39:50,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 20:39:50,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 20:39:50,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 20:39:50,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 20:39:50,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 20:39:50,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 20:39:50,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:39:50,868 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 20:39:50,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:39:50,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:50,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:50,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:50,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:50,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:51,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:51,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:51,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:51,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:51,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:51,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:51,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:51,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:51,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:51,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:51,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:51,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:51,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:51,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:39:51,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:39:51,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:51,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:39:51,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:39:51,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:39:51,726 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 20:39:51,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:39:51,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:51,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:39:51,760 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 20:39:51,760 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 20:39:51,763 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 20:39:51,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:39:52,171 | test | DEBUG | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.24 % 2021-12-15 20:39:52,171 | test | INFO | pool-3-thread-28 | [task:check:385] Rebalance - status: running, progress: 80.24 2021-12-15 20:39:52,194 | infra | DEBUG | pool-3-thread-28 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:39:52,197 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:39:52,197 | infra | DEBUG | pool-3-thread-28 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:39:52,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 20:39:52,331 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:39:52,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 20:39:52,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 20:39:52,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 20:39:52,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 20:39:52,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 20:39:52,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 20:39:52,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 20:39:52,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:39:52,500 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 20:39:52,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:39:52,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:52,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:52,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:52,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:39:52,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:52,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:52,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:52,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:39:52,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:52,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:52,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:52,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:39:52,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:53,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:53,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:39:53,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:53,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:53,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:39:53,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:53,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:53,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:39:53,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:53,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:53,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:39:53,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:39:53,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:53,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:39:53,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:39:53,615 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 20:39:53,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:39:53,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:39:53,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:39:53,667 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 20:39:53,668 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 20:39:53,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 20:39:53,674 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 20:40:02,207 | test | ERROR | pool-3-thread-28 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'e2efa055e792707c94d3520d1eab0b4e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=5971922df7f8718065c9d1be817aaa8b', u'status': u'notRunning'} - rebalance failed 2021-12-15 20:40:02,240 | test | INFO | pool-3-thread-28 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639629601589L, u'shortText': u'message', u'serverTime': u'2021-12-15T20:40:01.589Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 19:39:41.758519 165245 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639629601576L, u'shortText': u'message', u'serverTime': u'2021-12-15T20:40:01.576Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.22866.157>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.110.64\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.13900.152>,\n {update_vbucket_state,770,replica,\n passive,\'ns_1@172.23.110.68\'}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4,\n [{file,"src/janitor_agent.erl"},\n {line,377}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = 3655eaa78bf6cc0efb600beb6e7ec110'} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639629601557L, u'shortText': u'message', u'serverTime': u'2021-12-15T20:40:01.557Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639629601548L, u'shortText': u'message', u'serverTime': u'2021-12-15T20:40:01.548Z', u'text': u'Worker <0.23422.157> (for action {move,\n {770,\n [\'ns_1@172.23.110.68\',\n \'ns_1@172.23.110.64\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.65\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.22866.157>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.110.64\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.13900.152>,\n {update_vbucket_state,\n 770,\n replica,\n passive,\n \'ns_1@172.23.110.68\'}},\n infinity]}}}}]},\n [{janitor_agent,\n bulk_set_vbucket_state,\n 4,\n [{file,\n "src/janitor_agent.erl"},\n {line,\n 377}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 211}]}]}}}'} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639626232107L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:43:52.107Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639626231960L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:43:51.960Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639626231740L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:43:51.740Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 3655eaa78bf6cc0efb600beb6e7ec110"} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639626180595L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:43:00.595Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 170 seconds.'} 2021-12-15 20:40:02,240 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973244L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.244Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30976.151>,\n {{dcp_wait_for_data_move_failed,\n "GleamBookUsers0",967,\'ns_1@172.23.110.68\',\n [\'ns_1@172.23.110.65\',\'ns_1@172.23.110.64\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,451}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,211}]}]}}}}.\nRebalance Operation Id = b87a1daffb3d7d1d314234cf7fc2d051'} 2021-12-15 20:40:02,242 | test | ERROR | pool-3-thread-28 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639625973210L, u'shortText': u'message', u'serverTime': u'2021-12-15T19:39:33.210Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 18:37:30.702273 159651 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 20:40:02,242 | test | ERROR | pool-3-thread-28 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'e2efa055e792707c94d3520d1eab0b4e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=5971922df7f8718065c9d1be817aaa8b', u'status': u'notRunning'} - rebalance failed 2021-12-15 20:40:53,687 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 20:40:53,687 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 20:40:53,697 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:40:53,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:40:53,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:40:55,716 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:40:55,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:40:55,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:40:57,734 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:40:57,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:40:57,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:40:59,755 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:40:59,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:40:59,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:01,773 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:01,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:01,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:03,792 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:03,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:03,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:05,809 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:05,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:05,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:07,828 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:07,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:07,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:09,848 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:09,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:09,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:11,869 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:11,877 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:11,877 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:13,894 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:13,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:13,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:15,918 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:15,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:15,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:17,940 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:17,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:17,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:19,960 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:19,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:19,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:21,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:21,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:21,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:24,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:24,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:24,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:26,022 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:26,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:26,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:28,042 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:28,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:28,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:30,061 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:30,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:30,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:32,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:32,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:32,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:34,095 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:34,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:34,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:36,112 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:36,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:36,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:38,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:38,137 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:38,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:40,148 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:40,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:40,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:42,165 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:42,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:42,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:44,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:44,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:44,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:46,201 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:46,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:46,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:48,220 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:48,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:48,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:50,236 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:50,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:50,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:52,256 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:52,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:52,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:54,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:54,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:54,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:56,292 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:56,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:56,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:41:58,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:41:58,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:41:58,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:00,329 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:00,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:00,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:02,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:02,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:02,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:04,380 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:04,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:04,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:06,398 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:06,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:06,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:08,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:08,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:08,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:10,433 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:10,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:10,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:12,453 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:12,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:12,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:14,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:14,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:14,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:16,496 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:16,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:16,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:18,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:18,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:18,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:20,538 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:20,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:20,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:22,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:22,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:22,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:24,579 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:24,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:24,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:26,601 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:26,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:26,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:28,622 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:28,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:28,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:30,644 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:30,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:30,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:32,663 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:32,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:32,668 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:34,680 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:34,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:34,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:36,700 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:36,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:36,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:38,719 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:38,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:38,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:40,737 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:40,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:40,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:42,757 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:42,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:42,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:44,779 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:44,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:44,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:51,789 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error timed out 2021-12-15 20:42:51,789 | test | ERROR | MainThread | [rest_client:is_ns_server_running:79] Server 172.23.110.64:8091 is unavailable 2021-12-15 20:42:51,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:51,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:54,812 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:54,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:54,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:56,832 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:56,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:56,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:42:58,851 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:42:58,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:42:58,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:00,881 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:00,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:00,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:02,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:02,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:02,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:04,921 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:04,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:04,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:06,941 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:06,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:06,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:08,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:08,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:08,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:10,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:10,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:10,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:13,002 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:13,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:13,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:15,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:15,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:15,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:17,046 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:17,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:17,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:19,065 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:19,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:19,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:21,088 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:21,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:21,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:23,108 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:23,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:23,117 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:25,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:25,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:25,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:27,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:27,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:27,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:29,171 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:29,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:29,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:31,193 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:31,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:31,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:33,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:33,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:33,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:35,233 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:35,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:35,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:37,253 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:37,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:37,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:39,275 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:39,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:39,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:41,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:41,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:41,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:43,321 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:43,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:43,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:45,339 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:45,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:45,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:47,361 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:47,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:47,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:49,381 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:49,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:49,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:51,401 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:51,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:51,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:53,423 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:53,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:53,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:55,441 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:55,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:55,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:57,464 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:57,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:57,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:43:59,486 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:43:59,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:43:59,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:01,506 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:01,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:01,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:03,523 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:03,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:03,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:05,545 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:05,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:05,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:07,566 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:07,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:07,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:09,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:09,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:09,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:11,608 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 20:44:11,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 20:44:11,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 20:44:13,627 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 20:44:13,638 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 20:44:13,638 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 20:44:13,647 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 20:44:13,657 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-15 20:44:13,658 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 20:44:13,668 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-15 20:44:13,670 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 20:44:13,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:44:13,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 20:44:14,125 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:44:14,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 20:44:14,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 20:44:14,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 20:44:14,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 20:44:14,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 20:44:14,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 20:44:14,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 20:44:14,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:44:14,331 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 20:44:14,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:44:14,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:14,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:14,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:14,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:14,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:14,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:14,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:14,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:14,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:14,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:14,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:14,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:14,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:14,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:14,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:14,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:14,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:14,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:14,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:14,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:14,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:15,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:15,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:15,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:15,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:15,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:15,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:44:15,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:44:15,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:44:15,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:44:15,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 20:44:15,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 20:44:15,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:15,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 20:44:15,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 20:44:15,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:44:15,676 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 20:44:15,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:44:15,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:15,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:44:15,727 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 20:44:15,727 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 20:44:15,730 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 20:44:15,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:44:16,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 20:44:16,104 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:44:16,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 20:44:16,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 20:44:16,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 20:44:16,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 20:44:16,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 20:44:16,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 20:44:16,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 20:44:16,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:44:16,227 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 20:44:16,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:44:16,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:16,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:16,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:16,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:16,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:16,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:16,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:16,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:16,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:16,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:16,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:16,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:16,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:16,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:16,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:16,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:16,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:16,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:16,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:17,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:44:17,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 20:44:17,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:17,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:17,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:44:17,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 20:44:17,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:44:17,128 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 20:44:17,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:44:17,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:17,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:44:17,164 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 20:44:17,164 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 20:44:17,167 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 20:44:18,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 20:44:18,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 20:44:18,640 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 20:44:18,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 20:44:18,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 20:44:18,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 20:44:18,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 20:44:18,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 20:44:18,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 20:44:18,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 20:44:18,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 20:44:18,744 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 20:44:18,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 20:44:18,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:18,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:18,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:18,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 20:44:18,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:18,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:18,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:18,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 20:44:18,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:19,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 20:44:19,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:19,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 20:44:19,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:19,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 20:44:19,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:19,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 20:44:19,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:19,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 20:44:19,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 20:44:19,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:19,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 20:44:19,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 20:44:19,539 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 20:44:19,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 20:44:19,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 20:44:19,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 20:44:19,575 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 20:44:19,576 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 20:44:19,576 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639626231.67 2021-12-15 20:44:19,578 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 80 2021-12-15 20:44:19,608 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639629859.58 2021-12-15 20:44:19,609 | infra | INFO | pool-3-thread-29 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639629859.58' started 2021-12-15 20:44:19,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 20:44:19,621 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 20:44:19,634 | test | DEBUG | pool-3-thread-29 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-15 20:44:19,635 | test | DEBUG | pool-3-thread-29 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-15 20:44:19,688 | test | DEBUG | pool-3-thread-29 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 20:44:19,691 | test | INFO | pool-3-thread-29 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 3.37270249493 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.52684255887 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 7.23227479299 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 3.38190472214 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 20:44:19,700 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:44:19,700 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:44:19,717 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:44:19,720 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:44:19,720 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:44:29,730 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:44:29,730 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:44:29,746 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:44:29,750 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:44:29,750 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:44:39,762 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:44:39,762 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:44:39,778 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:44:39,780 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:44:39,780 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:44:49,792 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:44:49,792 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:44:49,812 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:44:49,815 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:44:49,815 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:44:59,823 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:44:59,825 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:44:59,841 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:44:59,842 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:44:59,842 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:09,854 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:09,854 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:45:09,869 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:45:09,872 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:45:09,872 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:19,882 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:19,882 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:45:19,898 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:45:19,901 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:45:19,901 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:29,911 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:29,911 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:45:29,928 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:45:29,930 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:45:29,931 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:39,940 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:39,941 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:45:39,956 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:45:39,957 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:45:39,957 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:49,967 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:49,967 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:45:49,982 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:45:49,983 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:45:49,983 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:45:59,993 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:45:59,993 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:00,009 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:00,010 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:00,010 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:46:10,020 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:10,020 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:10,036 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:10,039 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:10,039 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:46:19,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:19,634 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.0 2021-12-15 20:46:19,634 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639629859.58 2021-12-15 20:46:20,048 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:20,049 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:20,063 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:20,065 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:20,065 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:46:30,078 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:30,078 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:30,095 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:30,096 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:30,098 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:46:40,107 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:40,108 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:40,122 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:40,125 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:40,125 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:46:50,134 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:46:50,135 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:46:50,153 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:46:50,154 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:46:50,154 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:00,164 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:00,164 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:00,184 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:00,187 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:00,187 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:10,196 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:10,196 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:10,219 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:10,220 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:10,220 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:20,230 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:20,230 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:20,246 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:20,247 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:20,247 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:30,257 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:30,257 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:30,272 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:30,273 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:30,273 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:40,283 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:40,283 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:40,305 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:40,306 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:40,306 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:47:50,316 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:47:50,318 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:47:50,342 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:47:50,345 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:47:50,345 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:00,355 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 20:48:00,355 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 20:48:00,375 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:00,378 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:00,378 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:10,401 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.04 % 2021-12-15 20:48:10,401 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 1.04 2021-12-15 20:48:10,417 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:10,421 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:10,421 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:20,434 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.04 % 2021-12-15 20:48:20,434 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 1.04 2021-12-15 20:48:20,453 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:20,457 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:20,457 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:30,467 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.04 % 2021-12-15 20:48:30,467 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 21.04 2021-12-15 20:48:30,483 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:30,486 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:30,486 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:40,496 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.04 % 2021-12-15 20:48:40,496 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 21.04 2021-12-15 20:48:40,512 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:40,515 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:40,515 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:48:50,523 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.04 % 2021-12-15 20:48:50,525 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 21.04 2021-12-15 20:48:50,539 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:48:50,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:48:50,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:00,552 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.04 % 2021-12-15 20:49:00,552 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 21.04 2021-12-15 20:49:00,569 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:00,572 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:00,572 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:10,581 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.04 % 2021-12-15 20:49:10,582 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 21.04 2021-12-15 20:49:10,595 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:10,598 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:10,598 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:20,609 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:49:20,611 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:49:20,628 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:20,631 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:20,631 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:30,642 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:49:30,642 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:49:30,660 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:30,663 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:30,663 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:40,673 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:49:40,673 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:49:40,688 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:40,693 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:40,693 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:49:50,703 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:49:50,703 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:49:50,720 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:49:50,723 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:49:50,723 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:00,733 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:00,733 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:00,750 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:00,752 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:00,752 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:10,766 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:10,766 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:10,786 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:10,790 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:10,790 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:20,803 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:20,805 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:20,821 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:20,823 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:20,823 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:30,835 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:30,835 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:30,855 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:30,858 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:30,858 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:40,869 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:40,869 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:40,887 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:40,891 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:40,891 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:50:50,903 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:50:50,903 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:50:50,918 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:50:50,921 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:50:50,921 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:00,931 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:00,931 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:00,950 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:00,953 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:00,953 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:10,963 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:10,963 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:10,980 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:10,983 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:10,983 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:20,994 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:20,994 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:21,010 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:21,013 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:21,013 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:31,022 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:31,023 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:31,036 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:31,039 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:31,039 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:41,048 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:41,049 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:41,062 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:41,065 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:41,065 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:51:51,075 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:51:51,075 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:51:51,092 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:51:51,095 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:51:51,095 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:01,105 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:52:01,107 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:52:01,127 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:01,128 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:01,128 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:11,140 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:52:11,140 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:52:11,158 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:11,161 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:11,161 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:21,171 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.08 % 2021-12-15 20:52:21,171 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 22.08 2021-12-15 20:52:21,194 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:21,197 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:21,197 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:31,207 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.13 % 2021-12-15 20:52:31,207 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 23.13 2021-12-15 20:52:31,220 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:31,223 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:31,223 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:41,233 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.13 % 2021-12-15 20:52:41,233 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 23.13 2021-12-15 20:52:41,247 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:41,250 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:41,250 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:52:51,259 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.13 % 2021-12-15 20:52:51,260 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 23.13 2021-12-15 20:52:51,276 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:52:51,279 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:52:51,279 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:01,289 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.13 % 2021-12-15 20:53:01,289 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 43.13 2021-12-15 20:53:01,312 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:01,313 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:01,313 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:11,325 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.13 % 2021-12-15 20:53:11,325 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 43.13 2021-12-15 20:53:11,346 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:11,349 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:11,349 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:21,359 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.13 % 2021-12-15 20:53:21,359 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 43.13 2021-12-15 20:53:21,381 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:21,384 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:21,384 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:31,394 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.13 % 2021-12-15 20:53:31,394 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 43.13 2021-12-15 20:53:31,417 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:31,420 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:31,421 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:41,430 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:53:41,430 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:53:41,448 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:41,451 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:41,451 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:53:51,461 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:53:51,463 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:53:51,479 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:53:51,480 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:53:51,482 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:01,490 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:01,492 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:01,506 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:01,509 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:01,509 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:11,519 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:11,519 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:11,536 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:11,538 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:11,538 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:21,548 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:21,548 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:21,562 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:21,563 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:21,563 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:31,575 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:31,575 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:31,591 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:31,592 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:31,592 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:41,604 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:41,604 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:41,621 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:41,624 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:41,624 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:54:51,634 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:54:51,634 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:54:51,650 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:54:51,653 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:54:51,653 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:01,663 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:01,664 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:01,680 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:01,684 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:01,684 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:11,696 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:11,696 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:11,711 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:11,714 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:11,714 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:21,726 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:21,726 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:21,742 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:21,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:21,746 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:31,755 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:31,756 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:31,772 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:31,775 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:31,775 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:41,786 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:41,786 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:41,803 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:41,806 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:41,806 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:55:51,815 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:55:51,816 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:55:51,831 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:55:51,834 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:55:51,834 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:01,848 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:01,849 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:01,868 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:01,869 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:01,869 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:11,888 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:11,890 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:11,907 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:11,908 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:11,908 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:21,920 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:21,920 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:21,937 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:21,940 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:21,940 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:31,950 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:31,950 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:31,966 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:31,967 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:31,967 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:41,977 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:41,977 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:41,993 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:41,996 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:41,996 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:56:52,007 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.17 % 2021-12-15 20:56:52,009 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 44.17 2021-12-15 20:56:52,023 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:56:52,026 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:56:52,026 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:02,036 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-15 20:57:02,036 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 45.21 2021-12-15 20:57:02,051 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:02,052 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:02,052 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:12,062 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-15 20:57:12,062 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 45.21 2021-12-15 20:57:12,078 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:12,079 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:12,079 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:22,088 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.21 % 2021-12-15 20:57:22,089 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 55.21 2021-12-15 20:57:22,104 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:22,107 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:22,107 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:32,119 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.21 % 2021-12-15 20:57:32,119 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 55.21 2021-12-15 20:57:32,138 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:32,140 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:32,141 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:42,150 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.21 % 2021-12-15 20:57:42,150 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 55.21 2021-12-15 20:57:42,167 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:42,170 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:42,171 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:57:52,180 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.25 % 2021-12-15 20:57:52,180 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 56.25 2021-12-15 20:57:52,198 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:57:52,201 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:57:52,201 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:02,213 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.25 % 2021-12-15 20:58:02,214 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 56.25 2021-12-15 20:58:02,233 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:02,236 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:02,236 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:12,246 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.25 % 2021-12-15 20:58:12,246 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 56.25 2021-12-15 20:58:12,263 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:12,265 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:12,265 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:22,273 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.25 % 2021-12-15 20:58:22,275 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 56.25 2021-12-15 20:58:22,290 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:22,293 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:22,293 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:32,305 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:58:32,305 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:58:32,321 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:32,323 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:32,323 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:42,335 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:58:42,335 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:58:42,351 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:42,352 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:42,352 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:58:52,371 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:58:52,371 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:58:52,388 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:58:52,391 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:58:52,391 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:02,401 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:02,401 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:02,417 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:02,421 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:02,421 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:12,430 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:12,430 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:12,447 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:12,450 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:12,450 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:22,460 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:22,460 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:22,476 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:22,477 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:22,479 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:32,490 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:32,492 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:32,510 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:32,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:32,513 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:42,523 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:42,523 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:42,540 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:42,543 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:42,543 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 20:59:52,555 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 20:59:52,555 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 20:59:52,573 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 20:59:52,576 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 20:59:52,576 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:02,595 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:02,595 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:02,611 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:02,615 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:02,615 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:12,624 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:12,624 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:12,638 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:12,641 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:12,641 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:22,651 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:22,651 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:22,667 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:22,670 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:22,670 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:32,680 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:32,680 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:32,697 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:32,701 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:32,701 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:42,714 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:42,714 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:42,733 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:42,736 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:42,736 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:00:52,747 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:00:52,749 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:00:52,765 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:00:52,767 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:00:52,767 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:02,776 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:02,778 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:02,792 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:02,796 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:02,796 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:12,805 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:12,805 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:12,822 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:12,823 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:12,825 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:22,834 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:22,834 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:22,849 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:22,851 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:22,851 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:32,861 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:32,861 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:32,878 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:32,880 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:32,880 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:42,888 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:42,890 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:42,904 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:42,905 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:42,905 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:01:52,914 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.29 % 2021-12-15 21:01:52,915 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 57.29 2021-12-15 21:01:52,931 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:01:52,934 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:01:52,934 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:02,944 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.33 % 2021-12-15 21:02:02,944 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 58.33 2021-12-15 21:02:02,960 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:02,963 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:02,964 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:12,973 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.38 % 2021-12-15 21:02:12,973 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 59.38 2021-12-15 21:02:12,990 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:12,992 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:12,992 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:23,000 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.38 % 2021-12-15 21:02:23,002 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 59.38 2021-12-15 21:02:23,028 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:23,029 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:23,029 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:33,039 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.38 % 2021-12-15 21:02:33,039 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 59.38 2021-12-15 21:02:33,053 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:33,055 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:33,055 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:43,065 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.38 % 2021-12-15 21:02:43,065 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 59.38 2021-12-15 21:02:43,079 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:43,082 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:43,082 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:02:53,092 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.42 % 2021-12-15 21:02:53,092 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 60.42 2021-12-15 21:02:53,108 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:02:53,109 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:02:53,109 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:03,119 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.42 % 2021-12-15 21:03:03,121 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 60.42 2021-12-15 21:03:03,137 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:03,141 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:03,141 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:13,151 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.42 % 2021-12-15 21:03:13,151 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 60.42 2021-12-15 21:03:13,168 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:13,171 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:13,171 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:23,183 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.42 % 2021-12-15 21:03:23,183 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 60.42 2021-12-15 21:03:23,200 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:23,203 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:23,203 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:33,213 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:03:33,213 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:03:33,229 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:33,230 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:33,230 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:43,240 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:03:43,240 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:03:43,256 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:43,257 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:43,257 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:03:53,266 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:03:53,267 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:03:53,282 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:03:53,285 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:03:53,285 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:03,295 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:03,295 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:03,315 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:03,316 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:03,316 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:13,328 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:13,328 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:13,348 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:13,351 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:13,351 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:23,362 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:23,362 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:23,382 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:23,384 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:23,385 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:33,394 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:33,394 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:33,411 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:33,414 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:33,414 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:43,423 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:43,423 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:43,438 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:43,440 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:43,440 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:04:53,450 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:04:53,450 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:04:53,467 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:04:53,469 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:04:53,470 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:03,477 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:03,479 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:03,493 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:03,496 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:03,496 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:13,509 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:13,509 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:13,523 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:13,528 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:13,528 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:23,539 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:23,539 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:23,555 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:23,558 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:23,558 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:33,566 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:33,568 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:33,582 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:33,585 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:33,585 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:43,598 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:43,598 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:43,615 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:43,618 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:43,618 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:05:53,628 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:05:53,628 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:05:53,642 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:05:53,644 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:05:53,644 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:03,654 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:06:03,655 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:06:03,671 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:03,673 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:03,673 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:13,681 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:06:13,683 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:06:13,697 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:13,700 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:13,700 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:23,710 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:06:23,710 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:06:23,724 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:23,726 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:23,726 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:33,736 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:06:33,736 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:06:33,752 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:33,755 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:33,755 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:43,763 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.46 % 2021-12-15 21:06:43,765 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 61.46 2021-12-15 21:06:43,779 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:43,782 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:43,782 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:06:53,792 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.50 % 2021-12-15 21:06:53,792 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 62.5 2021-12-15 21:06:53,811 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:06:53,812 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:06:53,812 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:03,823 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.50 % 2021-12-15 21:07:03,823 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 62.5 2021-12-15 21:07:03,842 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:03,845 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:03,845 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:13,855 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.54 % 2021-12-15 21:07:13,855 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 63.54 2021-12-15 21:07:13,872 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:13,877 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:13,877 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:23,887 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.54 % 2021-12-15 21:07:23,887 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 63.54 2021-12-15 21:07:23,904 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:23,905 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:23,907 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:33,915 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.54 % 2021-12-15 21:07:33,915 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 63.54 2021-12-15 21:07:33,930 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:33,934 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:33,934 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:43,944 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.54 % 2021-12-15 21:07:43,944 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 63.54 2021-12-15 21:07:43,959 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:43,961 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:43,961 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:07:53,970 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.58 % 2021-12-15 21:07:53,971 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 64.58 2021-12-15 21:07:53,986 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:07:53,990 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:07:53,990 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:04,003 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.58 % 2021-12-15 21:08:04,003 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 64.58 2021-12-15 21:08:04,022 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:04,025 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:04,025 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:14,033 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.58 % 2021-12-15 21:08:14,035 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 64.58 2021-12-15 21:08:14,051 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:14,052 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:14,052 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:24,062 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.58 % 2021-12-15 21:08:24,062 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 64.58 2021-12-15 21:08:24,076 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:24,078 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:24,079 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:34,088 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:08:34,088 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:08:34,102 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:34,105 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:34,105 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:44,114 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:08:44,114 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:08:44,130 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:44,131 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:44,131 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:08:54,141 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:08:54,141 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:08:54,154 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:08:54,157 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:08:54,157 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:04,167 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:04,167 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:04,181 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:04,184 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:04,184 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:14,194 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:14,194 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:14,207 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:14,210 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:14,210 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:24,219 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:24,219 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:24,233 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:24,236 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:24,236 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:34,246 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:34,246 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:34,266 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:34,267 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:34,269 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:44,278 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:44,278 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:44,292 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:44,293 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:44,293 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:09:54,303 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:09:54,303 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:09:54,319 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:09:54,321 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:09:54,322 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:04,331 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:04,331 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:04,345 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:04,346 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:04,346 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:14,358 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:14,358 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:14,381 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:14,384 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:14,384 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:24,394 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:24,394 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:24,411 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:24,413 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:24,413 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:34,423 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:34,423 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:34,443 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:34,446 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:34,446 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:44,456 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:44,457 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:44,473 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:44,476 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:44,476 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:10:54,487 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:10:54,487 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:10:54,506 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:10:54,509 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:10:54,509 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:04,519 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:11:04,520 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:11:04,539 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:04,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:04,542 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:14,551 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:11:14,552 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:11:14,569 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:14,571 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:14,571 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:24,582 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:11:24,582 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:11:24,598 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:24,602 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:24,602 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:34,611 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.63 % 2021-12-15 21:11:34,612 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 65.63 2021-12-15 21:11:34,630 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:34,631 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:34,632 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:44,641 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.67 % 2021-12-15 21:11:44,641 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 66.67 2021-12-15 21:11:44,657 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:44,661 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:44,661 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:11:54,671 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.67 % 2021-12-15 21:11:54,671 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 66.67 2021-12-15 21:11:54,686 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:11:54,688 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:11:54,688 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:04,697 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.67 % 2021-12-15 21:12:04,698 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 66.67 2021-12-15 21:12:04,713 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:04,716 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:04,716 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:14,726 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.71 % 2021-12-15 21:12:14,726 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 67.71 2021-12-15 21:12:14,740 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:14,743 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:14,743 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:24,753 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.71 % 2021-12-15 21:12:24,753 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 67.71 2021-12-15 21:12:24,767 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:24,769 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:24,769 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:34,779 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.71 % 2021-12-15 21:12:34,779 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 67.71 2021-12-15 21:12:34,795 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:34,798 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:34,798 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:44,808 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.75 % 2021-12-15 21:12:44,808 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 68.75 2021-12-15 21:12:44,823 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:44,826 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:44,826 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:12:54,836 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.75 % 2021-12-15 21:12:54,836 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 68.75 2021-12-15 21:12:54,868 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:12:54,871 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:12:54,871 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:04,882 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.75 % 2021-12-15 21:13:04,882 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 68.75 2021-12-15 21:13:04,903 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:04,907 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:04,907 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:14,917 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.75 % 2021-12-15 21:13:14,917 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 68.75 2021-12-15 21:13:14,934 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:14,937 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:14,938 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:24,948 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:13:24,948 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:13:24,963 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:24,967 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:24,967 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:34,980 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:13:34,980 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:13:35,000 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:35,003 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:35,003 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:45,015 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:13:45,015 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:13:45,032 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:45,036 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:45,036 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:13:55,046 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:13:55,048 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:13:55,063 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:13:55,066 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:13:55,066 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:05,075 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:05,076 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:05,095 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:05,096 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:05,096 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:15,111 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:15,111 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:15,131 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:15,134 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:15,134 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:25,145 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:25,147 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:25,161 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:25,164 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:25,164 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:35,173 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:35,173 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:35,187 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:35,188 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:35,188 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:45,198 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:45,200 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:45,213 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:45,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:45,216 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:14:55,224 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:14:55,224 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:14:55,240 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:14:55,242 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:14:55,242 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:05,253 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:05,253 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:05,273 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:05,276 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:05,276 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:15,286 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:15,288 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:15,302 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:15,305 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:15,305 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:25,315 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:25,315 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:25,329 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:25,332 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:25,332 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:35,342 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:35,342 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:35,357 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:35,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:35,358 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:45,369 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:45,369 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:45,388 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:45,388 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:45,390 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:15:55,398 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:15:55,398 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:15:55,414 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:15:55,417 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:15:55,417 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:05,427 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.79 % 2021-12-15 21:16:05,427 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 69.79 2021-12-15 21:16:05,444 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:05,447 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:05,447 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:15,457 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.83 % 2021-12-15 21:16:15,457 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 70.83 2021-12-15 21:16:15,473 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:15,476 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:15,476 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:25,486 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.92 % 2021-12-15 21:16:25,486 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 72.92 2021-12-15 21:16:25,503 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:25,505 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:25,505 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:35,515 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.92 % 2021-12-15 21:16:35,515 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 72.92 2021-12-15 21:16:35,532 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:35,533 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:35,533 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:45,542 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.96 % 2021-12-15 21:16:45,543 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 73.96 2021-12-15 21:16:45,559 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:45,562 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:45,562 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:16:55,572 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.96 % 2021-12-15 21:16:55,572 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 73.96 2021-12-15 21:16:55,588 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:16:55,592 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:16:55,592 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:17:05,602 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.96 % 2021-12-15 21:17:05,602 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 73.96 2021-12-15 21:17:05,625 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:17:05,627 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:17:05,627 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:17:15,638 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.96 % 2021-12-15 21:17:15,638 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 73.96 2021-12-15 21:17:15,655 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:17:15,658 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:17:15,658 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:17:25,667 | test | DEBUG | pool-3-thread-29 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.00 % 2021-12-15 21:17:25,668 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: running, progress: 75.0 2021-12-15 21:17:25,684 | infra | DEBUG | pool-3-thread-29 | [task:check:412] Wait before next rebalance progress check 2021-12-15 21:17:25,687 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 21:17:25,687 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 21:17:35,697 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-15 21:17:35,698 | infra | DEBUG | pool-3-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-15 21:17:40,707 | test | INFO | pool-3-thread-29 | [task:check:385] Rebalance - status: none, progress: 100 2021-12-15 21:17:40,746 | test | INFO | pool-3-thread-29 | [task:check:444] Rebalance completed with progress: 100% in 2001.05800009 sec 2021-12-15 21:17:40,747 | infra | INFO | pool-3-thread-29 | [task:complete_task:105] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639629859.58' completed 2021-12-15 21:17:40,782 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 49922.4, u'diskFetches': 229877978, u'dataUsed': 8061417600729L, u'itemCount': 3999914830L, u'quotaPercentUsed': 78.1271708458662, u'storageTotals': {u'hdd': {u'usedByData': 11886881036819L, u'used': 12124039938044L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 23506178424180L}, u'ram': {u'usedByData': 251665232784L, u'used': 801467146240L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 11886881036819L, u'vbActiveNumNonResident': 3884102054L, u'memUsed': 251665232784L} 2021-12-15 21:17:40,986 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 3999914830 | 300.00 GiB | 234.38 GiB | 10.81 TiB | 2.89938537827 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-15 21:17:40,986 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-15 21:17:41,194 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 2.89938537827 2021-12-15 21:17:41,196 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:17:41,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:17:41,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:17:41,625 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:17:41,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:17:41,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:17:41,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:17:41,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:17:41,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:17:41,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:17:41,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:17:41,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 21:17:42,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 21:17:42,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 21:17:42,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 21:17:42,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 21:17:47,545 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 447 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 451 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-504 is 455 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 433 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 507 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 417 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 469 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-793 is 397 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 431 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 494 2021-12-15 21:17:47,546 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-433 is 455 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-361 is 433 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 458 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 434 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 484 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-802 is 496 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 449 2021-12-15 21:17:47,548 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-370 is 384 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-226 is 451 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 422 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 481 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 381 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-731 is 431 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 397 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 425 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 438 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 480 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 433 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 433 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-444 is 426 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-372 is 367 2021-12-15 21:17:47,549 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 412 2021-12-15 21:17:47,551 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 453 2021-12-15 21:17:47,551 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 424 2021-12-15 21:17:47,551 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 389 2021-12-15 21:17:47,551 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 497 2021-12-15 21:17:47,551 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-373 is 437 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 426 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 405 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 377 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-806 is 382 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 395 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 457 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-374 is 385 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 451 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 432 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 374 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-735 is 425 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-447 is 440 2021-12-15 21:17:47,552 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-375 is 448 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-231 is 427 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 457 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 362 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-736 is 454 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 427 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-376 is 411 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 429 2021-12-15 21:17:47,553 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 433 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-809 is 439 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 393 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-377 is 463 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 420 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 457 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-810 is 430 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 450 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 445 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 507 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 446 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-739 is 368 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 424 2021-12-15 21:17:47,555 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 382 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 408 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 400 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-794 is 447 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 451 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 479 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 445 2021-12-15 21:17:47,556 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 470 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-218 is 427 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-74 is 463 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 450 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-812 is 453 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-740 is 450 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 442 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 506 2021-12-15 21:17:47,558 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 516 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 498 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 423 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 436 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-381 is 373 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 453 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 517 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 434 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-742 is 441 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-454 is 411 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 487 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 406 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 448 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-815 is 444 2021-12-15 21:17:47,559 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-455 is 431 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 422 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-239 is 370 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 437 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-816 is 507 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 458 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-384 is 418 2021-12-15 21:17:47,561 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 435 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 487 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-817 is 378 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-457 is 448 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-385 is 425 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-241 is 379 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 407 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 419 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-746 is 391 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 472 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-386 is 427 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 450 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-26 is 395 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-819 is 382 2021-12-15 21:17:47,562 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-459 is 458 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-387 is 452 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 478 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 453 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 421 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-820 is 428 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-748 is 387 2021-12-15 21:17:47,563 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-460 is 324 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-388 is 460 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 500 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-172 is 425 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-28 is 423 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-821 is 377 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-749 is 428 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 413 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 482 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 465 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 414 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 454 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 463 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-723 is 418 2021-12-15 21:17:47,565 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 396 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 505 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-363 is 456 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 490 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-75 is 480 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 498 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-822 is 419 2021-12-15 21:17:47,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 413 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 432 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-390 is 425 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-246 is 511 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-174 is 380 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 471 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-823 is 418 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 412 2021-12-15 21:17:47,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 316 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-247 is 342 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 436 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 406 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 462 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 429 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 466 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 434 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 455 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 409 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 397 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-753 is 376 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-465 is 401 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-393 is 365 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 449 2021-12-15 21:17:47,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-177 is 465 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 411 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 407 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-754 is 463 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 458 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-394 is 420 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 449 2021-12-15 21:17:47,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 431 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 416 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-827 is 462 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-683 is 474 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 453 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-395 is 463 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 467 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 447 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 435 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 412 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 388 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-684 is 455 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 450 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-396 is 417 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-252 is 462 2021-12-15 21:17:47,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 436 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 396 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-469 is 432 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-397 is 333 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 460 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-181 is 383 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 498 2021-12-15 21:17:47,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-830 is 414 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 403 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 404 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 442 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-398 is 416 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 462 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 466 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-38 is 421 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 486 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 458 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-471 is 488 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-399 is 382 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-255 is 397 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 439 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 495 2021-12-15 21:17:47,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-796 is 464 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-724 is 375 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-508 is 481 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 394 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-364 is 438 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 443 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 442 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 416 2021-12-15 21:17:47,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 396 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-760 is 390 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 414 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 463 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 427 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 435 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-40 is 440 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-833 is 478 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-761 is 430 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-473 is 459 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-401 is 430 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 384 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 509 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 377 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-762 is 358 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 329 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 433 2021-12-15 21:17:47,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 430 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 406 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-835 is 403 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-691 is 444 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 411 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-403 is 424 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-187 is 457 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 509 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 403 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-692 is 415 2021-12-15 21:17:47,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 455 2021-12-15 21:17:47,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 439 2021-12-15 21:17:47,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 475 2021-12-15 21:17:47,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 462 2021-12-15 21:17:47,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-837 is 382 2021-12-15 21:17:47,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 409 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-405 is 361 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 426 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 422 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-838 is 398 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 473 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 432 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 426 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 467 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 471 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 437 2021-12-15 21:17:47,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-695 is 465 2021-12-15 21:17:47,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-479 is 358 2021-12-15 21:17:47,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 441 2021-12-15 21:17:47,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-191 is 471 2021-12-15 21:17:47,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-47 is 400 2021-12-15 21:17:47,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 443 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-696 is 446 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 430 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-408 is 476 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-192 is 475 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-48 is 508 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 445 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-697 is 433 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-481 is 419 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 383 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 464 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 428 2021-12-15 21:17:47,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 428 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 437 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 379 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-365 is 436 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 481 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 485 2021-12-15 21:17:47,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 395 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 488 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 472 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 460 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 467 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-410 is 465 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 480 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 551 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-843 is 409 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-699 is 438 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 427 2021-12-15 21:17:47,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-411 is 384 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 416 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-51 is 415 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 438 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-772 is 391 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-700 is 486 2021-12-15 21:17:47,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-484 is 409 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-412 is 443 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-196 is 463 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-52 is 468 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 457 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-773 is 420 2021-12-15 21:17:47,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 443 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-485 is 424 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 413 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 492 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 424 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 431 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-774 is 458 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-702 is 436 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 458 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-414 is 428 2021-12-15 21:17:47,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 434 2021-12-15 21:17:47,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 447 2021-12-15 21:17:47,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 495 2021-12-15 21:17:47,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 468 2021-12-15 21:17:47,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-415 is 429 2021-12-15 21:17:47,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 455 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 467 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 519 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 474 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 429 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 399 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 432 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 462 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-344 is 444 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 446 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 470 2021-12-15 21:17:47,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 432 2021-12-15 21:17:47,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-705 is 400 2021-12-15 21:17:47,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 462 2021-12-15 21:17:47,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 442 2021-12-15 21:17:47,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 459 2021-12-15 21:17:47,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 506 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 450 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-706 is 449 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 467 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 428 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 413 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 450 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 463 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 391 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 386 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 400 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-491 is 449 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-419 is 428 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 458 2021-12-15 21:17:47,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 410 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 432 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 409 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-726 is 472 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 476 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 404 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-366 is 443 2021-12-15 21:17:47,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-222 is 451 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 453 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-6 is 486 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-852 is 426 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 454 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-708 is 425 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 450 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-420 is 421 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 420 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 493 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 421 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-709 is 457 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 440 2021-12-15 21:17:47,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 430 2021-12-15 21:17:47,602 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 458 2021-12-15 21:17:47,602 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-205 is 409 2021-12-15 21:17:47,602 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-61 is 473 2021-12-15 21:17:47,602 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-782 is 451 2021-12-15 21:17:47,602 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 435 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 467 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-422 is 460 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 424 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 417 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 479 2021-12-15 21:17:47,604 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 410 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-711 is 319 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 405 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-423 is 376 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 439 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-207 is 439 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-63 is 419 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-496 is 464 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-424 is 471 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-352 is 430 2021-12-15 21:17:47,605 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 430 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-64 is 408 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-785 is 459 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 457 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-425 is 379 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 425 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-65 is 345 2021-12-15 21:17:47,607 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 475 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-498 is 478 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-426 is 437 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-354 is 392 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 422 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 476 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 400 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-715 is 449 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 334 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-355 is 477 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 447 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 494 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 429 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 403 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-428 is 312 2021-12-15 21:17:47,608 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 433 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 428 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 403 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-789 is 373 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-717 is 374 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 475 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 467 2021-12-15 21:17:47,609 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-357 is 419 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-213 is 421 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 420 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 425 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 418 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 481 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-367 is 414 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 461 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-79 is 519 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 502 2021-12-15 21:17:47,611 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-790 is 499 2021-12-15 21:17:47,612 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-718 is 434 2021-12-15 21:17:47,612 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 464 2021-12-15 21:17:47,612 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 431 2021-12-15 21:17:47,612 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-358 is 403 2021-12-15 21:17:47,612 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 424 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 439 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 428 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 458 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 437 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-359 is 414 2021-12-15 21:17:47,614 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 431 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 497 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 445 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-728 is 314 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 450 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 388 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-368 is 409 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 442 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 419 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 423 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-801 is 315 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-729 is 423 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 420 2021-12-15 21:17:47,615 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 485 2021-12-15 21:17:47,617 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 445 2021-12-15 21:17:47,617 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 461 2021-12-15 21:17:47,617 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:17:47,619 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:17:47,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:17:47,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:17:48,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:17:48,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:17:48,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:17:48,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:17:48,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:17:48,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:17:48,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:17:48,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:17:48,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 21:17:48,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 21:17:48,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 21:17:48,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 21:17:49,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 21:17:51,696 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-576 is 320 2021-12-15 21:17:51,696 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 324 2021-12-15 21:17:51,696 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 306 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-514 is 342 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-82 is 378 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-875 is 304 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 348 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 301 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-371 is 309 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-83 is 429 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 310 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-84 is 406 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-877 is 351 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-517 is 322 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-373 is 300 2021-12-15 21:17:51,697 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-157 is 309 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 500 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-1022 is 301 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-590 is 328 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-518 is 308 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 304 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-879 is 327 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-87 is 337 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-952 is 314 2021-12-15 21:17:51,698 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-592 is 311 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-160 is 313 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-593 is 332 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-377 is 315 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-89 is 308 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-378 is 301 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-162 is 335 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-90 is 321 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-955 is 310 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-595 is 303 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 303 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-578 is 329 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-362 is 308 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-74 is 340 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-884 is 348 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-92 is 311 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-957 is 304 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-597 is 333 2021-12-15 21:17:51,700 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-165 is 342 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-93 is 334 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-526 is 308 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-382 is 307 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-167 is 331 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-95 is 306 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-888 is 301 2021-12-15 21:17:51,701 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-528 is 301 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-384 is 316 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-98 is 300 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-387 is 315 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-100 is 313 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-965 is 326 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-893 is 318 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-101 is 325 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-966 is 301 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-534 is 308 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-103 is 312 2021-12-15 21:17:51,703 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-968 is 322 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-896 is 319 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-536 is 325 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-392 is 301 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-104 is 305 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-898 is 321 2021-12-15 21:17:51,704 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 334 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-107 is 318 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-972 is 333 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-396 is 301 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-973 is 323 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-397 is 322 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-902 is 313 2021-12-15 21:17:51,706 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-542 is 301 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-399 is 302 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-1012 is 307 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-76 is 329 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-544 is 335 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-400 is 306 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-403 is 310 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-115 is 310 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 311 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 302 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-981 is 315 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 314 2021-12-15 21:17:51,707 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-118 is 348 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-911 is 300 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 302 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 333 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-941 is 324 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-149 is 307 2021-12-15 21:17:51,709 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-915 is 336 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-555 is 317 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 309 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-556 is 324 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 327 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-413 is 300 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-125 is 300 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-918 is 318 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 322 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-343 is 318 2021-12-15 21:17:51,710 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-127 is 323 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-993 is 304 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 311 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-345 is 335 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-129 is 311 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-130 is 311 2021-12-15 21:17:51,711 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 330 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-347 is 302 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-582 is 320 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-366 is 330 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-348 is 336 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-132 is 318 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-565 is 316 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-349 is 310 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-133 is 337 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-998 is 308 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-926 is 321 2021-12-15 21:17:51,713 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-350 is 335 2021-12-15 21:17:51,714 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-134 is 305 2021-12-15 21:17:51,714 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-64 is 399 2021-12-15 21:17:51,714 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-929 is 325 2021-12-15 21:17:51,714 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-857 is 334 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-353 is 314 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-137 is 333 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-426 is 320 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-354 is 305 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-138 is 308 2021-12-15 21:17:51,716 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-427 is 309 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-355 is 318 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-1004 is 312 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-933 is 310 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-357 is 306 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-141 is 310 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-1015 is 317 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-367 is 320 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-79 is 309 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-574 is 302 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-358 is 330 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-142 is 305 2021-12-15 21:17:51,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-944 is 326 2021-12-15 21:17:51,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-584 is 325 2021-12-15 21:17:51,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 326 2021-12-15 21:17:51,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-585 is 308 2021-12-15 21:17:51,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-513 is 300 2021-12-15 21:17:51,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-81 is 475 2021-12-15 21:17:51,719 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:17:51,723 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:17:51,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:17:52,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:17:52,403 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:17:52,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:17:52,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:17:52,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:17:52,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:17:52,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:17:52,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:17:52,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:17:52,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-15 21:17:53,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-15 21:17:53,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-15 21:17:53,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-15 21:17:53,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-15 21:17:56,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-936 is 324 2021-12-15 21:17:56,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-720 is 325 2021-12-15 21:17:56,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-288 is 329 2021-12-15 21:17:56,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.0/kvstore-216 is 372 2021-12-15 21:17:56,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-217 is 453 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.1/kvstore-145 is 308 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-874 is 334 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-730 is 319 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-658 is 329 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-514 is 302 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.10/kvstore-226 is 307 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.11/kvstore-227 is 444 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-876 is 313 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-660 is 315 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-300 is 305 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.12/kvstore-228 is 450 2021-12-15 21:17:56,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.13/kvstore-229 is 318 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-878 is 301 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-734 is 300 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-662 is 326 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.14/kvstore-302 is 304 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-735 is 329 2021-12-15 21:17:56,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-663 is 345 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-303 is 312 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.15/kvstore-231 is 438 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-880 is 306 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-736 is 346 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-664 is 312 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-592 is 315 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-304 is 301 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.16/kvstore-232 is 352 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-881 is 332 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-593 is 337 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.17/kvstore-233 is 419 2021-12-15 21:17:56,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-738 is 362 2021-12-15 21:17:56,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-666 is 312 2021-12-15 21:17:56,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-306 is 326 2021-12-15 21:17:56,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.18/kvstore-234 is 366 2021-12-15 21:17:56,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-883 is 323 2021-12-15 21:17:56,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-739 is 308 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-667 is 332 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-307 is 300 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.19/kvstore-235 is 313 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.2/kvstore-218 is 419 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-884 is 346 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-740 is 327 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-668 is 341 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.20/kvstore-524 is 313 2021-12-15 21:17:56,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-885 is 329 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-597 is 332 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-525 is 323 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-309 is 335 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.21/kvstore-237 is 311 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-886 is 301 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-742 is 360 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-670 is 301 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.22/kvstore-238 is 303 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-887 is 315 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-311 is 324 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.23/kvstore-239 is 448 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-888 is 340 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-744 is 319 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-672 is 331 2021-12-15 21:17:56,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-600 is 314 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.24/kvstore-240 is 433 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-889 is 303 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-313 is 335 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.25/kvstore-241 is 436 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-890 is 306 2021-12-15 21:17:56,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-746 is 306 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-530 is 315 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.26/kvstore-242 is 451 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-891 is 301 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-747 is 324 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-243 is 415 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.27/kvstore-171 is 428 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-676 is 302 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-532 is 307 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-316 is 331 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-244 is 402 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.28/kvstore-172 is 419 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-893 is 325 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-605 is 317 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-533 is 331 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.29/kvstore-245 is 398 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-723 is 329 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-579 is 326 2021-12-15 21:17:56,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.3/kvstore-219 is 305 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-894 is 334 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-318 is 342 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-246 is 362 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.30/kvstore-174 is 380 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-535 is 336 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-247 is 365 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.31/kvstore-175 is 384 2021-12-15 21:17:56,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-896 is 306 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-680 is 301 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-608 is 309 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-536 is 323 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.32/kvstore-176 is 418 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-897 is 310 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-753 is 315 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-681 is 336 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-321 is 331 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.33/kvstore-249 is 347 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-610 is 331 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-250 is 491 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.34/kvstore-178 is 425 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-899 is 318 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-683 is 326 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-611 is 312 2021-12-15 21:17:56,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-251 is 411 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.35/kvstore-179 is 330 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-900 is 335 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-684 is 310 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-612 is 356 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.36/kvstore-252 is 315 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-901 is 329 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-685 is 301 2021-12-15 21:17:56,737 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-541 is 318 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-325 is 333 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.37/kvstore-181 is 362 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-902 is 331 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-758 is 305 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-326 is 319 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-254 is 351 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.38/kvstore-182 is 368 2021-12-15 21:17:56,739 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-903 is 312 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-759 is 336 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-687 is 346 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-255 is 385 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.39/kvstore-183 is 398 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-868 is 339 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-292 is 309 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.4/kvstore-220 is 416 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-904 is 328 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-760 is 323 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-688 is 312 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.40/kvstore-184 is 307 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-905 is 316 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-761 is 308 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-689 is 321 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-617 is 300 2021-12-15 21:17:56,740 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-257 is 305 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.41/kvstore-185 is 413 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.42/kvstore-690 is 310 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.42/kvstore-186 is 436 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-763 is 324 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-547 is 303 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-331 is 317 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-259 is 347 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.43/kvstore-187 is 342 2021-12-15 21:17:56,742 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-621 is 306 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-549 is 330 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.45/kvstore-189 is 401 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-910 is 326 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.46/kvstore-190 is 395 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-911 is 317 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-767 is 329 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-695 is 324 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-335 is 342 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.47/kvstore-191 is 338 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-624 is 329 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-336 is 332 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.48/kvstore-192 is 439 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-913 is 324 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-553 is 331 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-337 is 331 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-265 is 339 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.49/kvstore-193 is 476 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-581 is 333 2021-12-15 21:17:56,743 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-293 is 310 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.5/kvstore-221 is 470 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-554 is 329 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-338 is 321 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-266 is 313 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.50/kvstore-194 is 352 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-915 is 310 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-627 is 322 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.51/kvstore-195 is 332 2021-12-15 21:17:56,744 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-916 is 311 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-628 is 314 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-268 is 314 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.52/kvstore-196 is 500 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-917 is 337 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-701 is 351 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-341 is 345 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-269 is 322 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.53/kvstore-197 is 479 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-270 is 334 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.54/kvstore-198 is 478 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-919 is 318 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-703 is 305 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-631 is 306 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-559 is 327 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-199 is 354 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.55/kvstore-127 is 301 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-920 is 308 2021-12-15 21:17:56,746 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-632 is 325 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.56/kvstore-200 is 339 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-705 is 326 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-273 is 314 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.57/kvstore-201 is 410 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.58/kvstore-562 is 325 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.58/kvstore-202 is 450 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.59/kvstore-923 is 306 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.59/kvstore-707 is 336 2021-12-15 21:17:56,747 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-870 is 328 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-654 is 343 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.6/kvstore-582 is 315 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-708 is 303 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-636 is 302 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-564 is 318 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-276 is 314 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.60/kvstore-204 is 367 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-709 is 311 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-565 is 304 2021-12-15 21:17:56,749 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-277 is 357 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.61/kvstore-205 is 434 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-854 is 324 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-710 is 321 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-566 is 303 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-278 is 308 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.62/kvstore-206 is 329 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-927 is 304 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-855 is 308 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-711 is 333 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-279 is 339 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.63/kvstore-207 is 370 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-856 is 341 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-712 is 311 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-568 is 330 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-280 is 321 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.64/kvstore-208 is 404 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-929 is 322 2021-12-15 21:17:56,750 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-713 is 318 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-569 is 305 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.65/kvstore-209 is 462 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-930 is 308 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-570 is 331 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.66/kvstore-210 is 415 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-931 is 353 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-715 is 341 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-643 is 318 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-283 is 324 2021-12-15 21:17:56,752 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.67/kvstore-211 is 457 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-644 is 312 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-572 is 313 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.68/kvstore-212 is 432 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-933 is 319 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-645 is 338 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.69/kvstore-285 is 342 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-727 is 325 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-583 is 322 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.7/kvstore-223 is 355 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-934 is 303 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-286 is 343 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.70/kvstore-214 is 418 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-935 is 307 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-863 is 321 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-719 is 342 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.71/kvstore-287 is 312 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-872 is 305 2021-12-15 21:17:56,753 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-584 is 336 2021-12-15 21:17:56,755 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.8/kvstore-224 is 465 2021-12-15 21:17:56,755 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-729 is 326 2021-12-15 21:17:56,755 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-657 is 318 2021-12-15 21:17:56,755 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-513 is 304 2021-12-15 21:17:56,755 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.66--/data/GleamBookUsers0/magma.9/kvstore-297 is 318 2021-12-15 21:17:56,755 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:17:56,755 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 11379256MB 2021-12-15 21:17:56,755 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 9734MB 2021-12-15 21:17:56,755 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 389856MB 2021-12-15 21:17:56,756 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 10988936MB 2021-12-15 21:17:56,757 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:17:56,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:17:57,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:17:57,115 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:17:57,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:17:57,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:17:57,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:17:57,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:17:57,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:17:57,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:17:57,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:17:57,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 21:17:57,239 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-15 21:17:57,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 21:17:57,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 21:17:58,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 21:17:58,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 21:17:58,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 21:17:59,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 21:17:59,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 21:17:59,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 21:18:00,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 21:18:00,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 21:18:00,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 21:18:01,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 21:18:01,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 21:18:01,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 21:18:02,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 21:18:02,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 21:18:03,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 21:18:03,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 21:18:03,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 21:18:04,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 21:18:04,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 21:18:04,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 21:18:05,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 21:18:05,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 21:18:05,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 21:18:06,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 21:18:06,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 21:18:06,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 21:18:07,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 21:18:07,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 21:18:07,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 21:18:08,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 21:18:08,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 21:18:09,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 21:18:09,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 21:18:09,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 21:18:10,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 21:18:10,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 21:18:10,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 21:18:11,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 21:18:11,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 21:18:11,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 21:18:12,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 21:18:12,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 21:18:12,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 21:18:13,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 21:18:13,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 21:18:13,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 21:18:14,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 21:18:14,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 21:18:14,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 21:18:15,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 21:18:15,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 21:18:15,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 21:18:16,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 21:18:16,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 21:18:17,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 21:18:17,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 21:18:17,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 21:18:18,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 21:18:18,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 21:18:18,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 21:18:19,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 21:18:19,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 21:18:19,802 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:18:19,806 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:18:19,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:18:20,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:18:20,165 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:18:20,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:18:20,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:18:20,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:18:20,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:18:20,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:18:20,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:18:20,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:18:20,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 21:18:20,285 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-15 21:18:20,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 21:18:20,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 21:18:21,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 21:18:21,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 21:18:21,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 21:18:22,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 21:18:22,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 21:18:22,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 21:18:23,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 21:18:23,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 21:18:23,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 21:18:24,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 21:18:24,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 21:18:25,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 21:18:25,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 21:18:25,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 21:18:26,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 21:18:26,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 21:18:26,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 21:18:27,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 21:18:27,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 21:18:27,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 21:18:28,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 21:18:28,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 21:18:28,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 21:18:29,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 21:18:29,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 21:18:29,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 21:18:30,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 21:18:30,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 21:18:30,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 21:18:31,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 21:18:31,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 21:18:32,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 21:18:32,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 21:18:32,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 21:18:33,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 21:18:33,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 21:18:33,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 21:18:34,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 21:18:34,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 21:18:34,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 21:18:35,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 21:18:35,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 21:18:35,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 21:18:36,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 21:18:36,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 21:18:37,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 21:18:37,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 21:18:37,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 21:18:38,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 21:18:38,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 21:18:38,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 21:18:39,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 21:18:39,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 21:18:39,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 21:18:40,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 21:18:40,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 21:18:40,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 21:18:41,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 21:18:41,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 21:18:41,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 21:18:42,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 21:18:42,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 21:18:43,010 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:18:43,013 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:18:43,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:18:43,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:18:43,463 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:18:43,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:18:43,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:18:43,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:18:43,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:18:43,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:18:43,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:18:43,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:18:43,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-15 21:18:43,621 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-15 21:18:43,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-15 21:18:43,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-15 21:18:44,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-15 21:18:44,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-15 21:18:45,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-15 21:18:45,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-15 21:18:45,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-15 21:18:46,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-15 21:18:46,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-15 21:18:47,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-15 21:18:47,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-15 21:18:47,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-15 21:18:48,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-15 21:18:48,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-15 21:18:48,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-15 21:18:49,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-15 21:18:49,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-15 21:18:50,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-15 21:18:50,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-15 21:18:51,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-15 21:18:51,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-15 21:18:51,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-15 21:18:52,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-15 21:18:52,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-15 21:18:53,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-15 21:18:53,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-15 21:18:53,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-15 21:18:54,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-15 21:18:54,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-15 21:18:55,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-15 21:18:55,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-15 21:18:55,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-15 21:18:56,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-15 21:18:56,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-15 21:18:56,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-15 21:18:57,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-15 21:18:57,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-15 21:18:58,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-15 21:18:58,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-15 21:18:58,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-15 21:18:59,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-15 21:18:59,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-15 21:19:00,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-15 21:19:00,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-15 21:19:01,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-15 21:19:01,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-15 21:19:01,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-15 21:19:02,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-15 21:19:02,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-15 21:19:02,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-15 21:19:03,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-15 21:19:03,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-15 21:19:03,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-15 21:19:04,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-15 21:19:04,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-15 21:19:04,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-15 21:19:05,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-15 21:19:05,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-15 21:19:05,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-15 21:19:06,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-15 21:19:06,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-15 21:19:07,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-15 21:19:07,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-15 21:19:07,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-15 21:19:08,059 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:19:08,059 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 890266.125, u'ReplayWALOffset': u'11404:4456448', u'NTablesCreated': 60173, u'RecentReadIOAmpSet': 0.18223144114017487, u'ReadIOAmp': 209.2388153076172, u'BlockCacheMisses': 5237666, u'NFlushes': 24451, u'RecentWriteAmp': 1.3860161304473877, u'NumSyncFlushes': 1857, u'TotalMemUsed': 138485457, u'NBloomFilterHits': 48784455, u'Fragmentation': 0.2874740660190582, u'NumNormalFlushes': 15281, u'RecentBlockCacheHitRatio': 78.78024291992188, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 32873, u'ReadIOAmp': 1.0149642135421437, u'MemtablesMemUsed': 6402360, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 453826694, u'CheckpointOverhead': 3604480, u'NItemsGet': 1320751, u'NWriteIOs': 83327, u'NReadBytes': 5767770692L, u'LevelStats': [{u'TargetSize': 62914560, u'LogicalSize': 12440826, u'SizeRatio': 0.19774160385131836, u'NWriteBytes': 112472064, u'BloomFilterMemUsed': 0, u'NumTables': 71, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1036288}, {u'TargetSize': 62914560, u'LogicalSize': 24056817, u'SizeRatio': 0.3823728084564209, u'NWriteBytes': 76615680, u'BloomFilterMemUsed': 0, u'NumTables': 16, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 659456}, {u'TargetSize': 647288721, u'LogicalSize': 506029339, u'SizeRatio': 0.7817675831246255, u'NWriteBytes': 499548160, u'BloomFilterMemUsed': 0, u'NumTables': 188, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 20918272}, {u'TargetSize': 32838542095L, u'LogicalSize': 19264158950L, u'SizeRatio': 0.5866325884465243, u'NWriteBytes': 158900224, u'BloomFilterMemUsed': 0, u'NumTables': 943, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 192950272}, {u'TargetSize': 129457744894L, u'LogicalSize': 65677084192L, u'SizeRatio': 0.5073244883554583, u'NWriteBytes': 18572972032L, u'BloomFilterMemUsed': 0, u'NumTables': 442, u'WastedSpace': 948211745, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 65677074432L}], u'NItemsGarbageCollected': 726160, u'NInternalFragmentationCompacts': 1, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 405879431, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 175584, u'NNonL0LocalCompacts': 4, u'NWriterCompacts': 0, u'NReadIOs': 1955895, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 515778, u'CompactWriter': {u'NDataBlockWrites': 366835, u'NWriteBytes': 830713856, u'NIndexBlockWrites': 12719, u'NBytesIncoming': 1480318731, u'NSyncs': 0, u'NInserts': 66756884, u'NWriteIOs': 10782}, u'NWriteBytes': 24094432417L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1340515, u'NReadBytes': 4283365713L, u'NDataBlockReads': 1320751, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3962253}, u'NBloomFilterMisses': 0, u'NItemsMerged': 633287, u'LogicalDiskSize': 65677084192L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7982480937551882, u'BufferMemUsed': 653574, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 46412925242L, u'NTableFiles': 1760, u'NFileCountCompacts': 522, u'TableObjectMemUsed': 453592, u'TreeSnapshotMemoryUsed': 223328, u'TotalDiskUsage': 73990410240L, u'BytesIncoming': 30184140251L, u'NItemsDel': 0, u'PendingCheckpointBytes': 55029760, u'NCompacts': 2190, u'NItemsPut': 23835189, u'NSyncs': 9174, u'CompactReader': {u'NReadIOs': 612873, u'NReadBytes': 1480855868, u'NDataBlockReads': 587941, u'NBytesOutgoing': 0, u'NIndexBlockReads': 24932}, u'ActiveDiskUsage': 65892638720L, u'NTablesDeleted': 21352, u'IterateReader': {u'NReadIOs': 2507, u'NReadBytes': 3549111, u'NDataBlockReads': 1358, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1149}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 6983727, u'NWriteBytes': 23263718561L, u'NIndexBlockWrites': 89351, u'NBytesIncoming': 30283802641L, u'NSyncs': 9174, u'NInserts': 24931164, u'NWriteIOs': 72545}}, u'NReadBytes': 1106265886167L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 233.0, u'NInternalFragmentationCompacts': 1, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 433854420, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 8365, u'NTxns': 167695, u'NWriteBytes': 33881329664L, u'NSegments': 1276, u'NSyncs': 167695, u'NLoggedBytes': 33881329664L, u'NActiveSegments': 1, u'NWrites': 262362, u'NInserts': 25308617, u'DiskUsed': 16703488}, u'NGets': 1982985, u'NReadBytesSet': 1765567551, u'NReadBytesCompact': 9901263617L, u'IndexResidentRatio': 0.04017243534326553, u'CompactQueueSize': 0, u'NSyncsPerSec': 5.0, u'BlockCacheHitRatio': 70.15728759765625, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1096361073439L, u'BloomFilterMemUsedRatio': 0.2679702341556549, u'NWriteBytes': 67629703408L, u'BytesOutgoing': 1861865066, u'RecentReadIOAmp': 2.18335223197937, u'WriteAmp': 2.2452375888824463, u'NBlocksCached': 25855, u'LogicalDataSize': 48104339604L, u'ActiveBloomFilterMemUsed': 46147064, u'NFileCountCompacts': 1810, u'NTableFiles': 2795, u'FlushQueueSize': 72, u'TableObjectMemUsed': 710272, u'BlockCacheQuota': 86078155, u'TotalDiskUsage': 76083097600L, u'NPurgerRuns': 32, u'ReadIOAmpSet': 0.0, u'NSyncs': 27522, u'NReadIOsSetPerSec': 61.46666717529297, u'NReadIOsPerSec': 444.20001220703125, u'NTablesDeleted': 48258, u'BloomFilterFPR': 0.013930108398199081, u'NBloomFilterHitsPerSec': 2133.2666015625, u'WriteCacheMemUsed': 8885160, u'RecentBloomFilterFPR': 0.016338512301445007, u'BytesPerRead': 551993.8125, u'BytesIncomingPerSec': 2162142.5, u'NReadIOsGetPerSec': 319.1333312988281, u'NGetStats': 14015, u'TotalIndexBlocksSize': 482189404, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 8835829760L, u'NReadIOsSet': 669966, u'ActiveDataSize': 48104339604L, u'NWriteBytesCompactPerSec': 298188.8125, u'NBloomFilterMissesPerSec': 35.43333435058594, u'NCommitBatches': 166370, u'NGetStatsComputed': 14015, u'NWriteIOs': 407589, u'NReadIOsGet': 415587419, u'RecentReadAmpGet': 6.604247093200684, u'WriteCacheQuota': 21519540, u'LSMTreeObjectMemUsed': 248640, u'NSetsPerSec': 337.29998779296875, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 7.066666603088379, u'ManagedFileObjectMemUsed': 274944, u'ReadAmpGet': 587.9027099609375, u'BlockCacheMemUsed': 86172384, u'NWriterCompacts': 0, u'NReadIOs': 419229121, u'WALBufferMemUsed': 65536, u'ReadAmp': 593.2225341796875, u'keyStats': {u'NItemsTransformed': 178176508, u'NTablesCreated': 15213, u'ReadIOAmp': 84.74317782995728, u'MemtablesMemUsed': 536888, u'NBloomFilterHits': 48781634, u'TotalIndexBlocksSize': 28350356, u'CheckpointOverhead': 30568448, u'NItemsGet': 4888254, u'NWriteIOs': 40639, u'NReadBytes': 1100475696755L, u'LevelStats': [{u'TargetSize': 62914560, u'LogicalSize': 23750734, u'SizeRatio': 0.377507750193278, u'NWriteBytes': 1245171712, u'BloomFilterMemUsed': 444552, u'NumTables': 150, u'WastedSpace': 5610, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 23490560}, {u'TargetSize': 62914560, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 62914560, u'LogicalSize': 50832690, u'SizeRatio': 0.8079638481140137, u'NWriteBytes': 2331033600L, u'BloomFilterMemUsed': 1689600, u'NumTables': 30, u'WastedSpace': 121856, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 50954240}, {u'TargetSize': 183514374, u'LogicalSize': 143908031, u'SizeRatio': 0.7841785243481799, u'NWriteBytes': 1597120512, u'BloomFilterMemUsed': 5163520, u'NumTables': 75, u'WastedSpace': 369770, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 144277504}, {u'TargetSize': 1835143808, u'LogicalSize': 1835143808, u'SizeRatio': 1.0, u'NWriteBytes': 2587365376L, u'BloomFilterMemUsed': 38847232, u'NumTables': 400, u'WastedSpace': 6212564, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1841332224}], u'NItemsGarbageCollected': 2238793, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 27969209, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 75168, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 417248432, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 1287620, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 265413, u'CompactWriter': {u'NDataBlockWrites': 2902438, u'NWriteBytes': 7999635456L, u'NIndexBlockWrites': 37454, u'NBytesIncoming': 11051408620L, u'NSyncs': 0, u'NInserts': 201972822, u'NWriteIOs': 18025}, u'NWriteBytes': 9550330489L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 414246178, u'NReadBytes': 1092077104007L, u'NDataBlockReads': 796402886, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1479798081}, u'NBloomFilterMisses': 688975, u'NItemsMerged': 0, u'LogicalDiskSize': 1835143808, u'TotalBloomFilterMemUsed': 39960456, u'WriteAmp': 8.652199258402993, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 46144904, u'LogicalDataSize': 1691235777, u'NTableFiles': 783, u'NFileCountCompacts': 619, u'TableObjectMemUsed': 194184, u'TreeSnapshotMemoryUsed': 127456, u'TotalDiskUsage': 2090622976, u'BytesIncoming': 1103803808, u'NItemsDel': 0, u'PendingCheckpointBytes': 16384, u'NCompacts': 1539, u'NItemsPut': 22409172, u'NSyncs': 9174, u'CompactReader': {u'NReadIOs': 3002254, u'NReadBytes': 8398592748L, u'NDataBlockReads': 2953289, u'NBytesOutgoing': 0, u'NIndexBlockReads': 48965}, u'ActiveDiskUsage': 2060054528, u'NTablesDeleted': 14872, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.013926956104381088, u'Writer': {u'NDataBlockWrites': 350218, u'NWriteBytes': 1550695033, u'NIndexBlockWrites': 15252, u'NBytesIncoming': 1299822334, u'NSyncs': 9174, u'NInserts': 22311559, u'NWriteIOs': 22614}}, u'NReadBytesCompactPerSec': 300401.71875, u'TableMetaMemUsed': 853539, u'NDeletes': 0, u'BlockCacheHitsPerSec': 865.0333251953125, u'BytesOutgoingPerSec': 134802.0625, u'NBloomFilterMisses': 689173, u'LogicalDiskSize': 67512406585L, u'NSets': 2756884, u'TotalBloomFilterMemUsed': 39962616, u'BufferMemUsed': 653574, u'RecentReadAmp': 8.832712173461914, u'NGetsPerSec': 146.1666717529297, u'NInserts': 19652288, u'NCommitBatchesPerSec': 203.43333435058594, u'FragmentationWithOverhead': 0.3677394390106201, u'NCompactsPerSec': 0.46666666865348816, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 1190667.875, u'TreeSnapshotMemoryUsed': 409472, u'BytesIncoming': 30121402411L, u'NCompacts': 4398, u'NumPersistentFlushes': 7313, u'NWriteIOsPerSec': 221.1666717529297, u'NInsertsPerSec': 1099.0, u'RecentBytesPerRead': 6090.76025390625, u'ActiveDiskUsage': 67953651712L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 160771.0, u'BlockCacheHits': 12313241, u'NWriteBytesPerSec': 2996764.5, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 12087, u'ReadIOAmp': 0.19318786588610964, u'MemtablesMemUsed': 1945912, u'NBloomFilterHits': 2821, u'TotalIndexBlocksSize': 12354, u'CheckpointOverhead': 1105920, u'NItemsGet': 3758, u'NWriteIOs': 21261, u'NReadBytes': 22418720, u'LevelStats': [{u'TargetSize': 62914560, u'LogicalSize': 1025229, u'SizeRatio': 0.016295576095581056, u'NWriteBytes': 81235968, u'BloomFilterMemUsed': 1200, u'NumTables': 102, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 835584}, {u'TargetSize': 62914560, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 62914560, u'LogicalSize': 178585, u'SizeRatio': 0.0028385321299235025, u'NWriteBytes': 4767744, u'BloomFilterMemUsed': 960, u'NumTables': 15, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 122880}], u'NItemsGarbageCollected': 487638, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 5780, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 24192, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 24794, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 72348, u'CompactWriter': {u'NDataBlockWrites': 669, u'NWriteBytes': 5480448, u'NIndexBlockWrites': 669, u'NBytesIncoming': 3611708, u'NSyncs': 0, u'NInserts': 35457, u'NWriteIOs': 669}, u'NWriteBytes': 103610838, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 726, u'NReadBytes': 603719, u'NDataBlockReads': 1134, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1134}, u'NBloomFilterMisses': 198, u'NItemsMerged': 0, u'LogicalDiskSize': 178585, u'TotalBloomFilterMemUsed': 2160, u'WriteAmp': 0.46108599001706396, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 2160, u'LogicalDataSize': 178585, u'NTableFiles': 252, u'NFileCountCompacts': 669, u'TableObjectMemUsed': 62496, u'TreeSnapshotMemoryUsed': 58688, u'TotalDiskUsage': 2064384, u'BytesIncoming': 224710445, u'NItemsDel': 0, u'PendingCheckpointBytes': 8192, u'NCompacts': 669, u'NItemsPut': 490978, u'NSyncs': 9174, u'CompactReader': {u'NReadIOs': 24068, u'NReadBytes': 21815001, u'NDataBlockReads': 12034, u'NBytesOutgoing': 0, u'NIndexBlockReads': 12034}, u'ActiveDiskUsage': 958464, u'NTablesDeleted': 12034, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.06558463067240808, u'Writer': {u'NDataBlockWrites': 11418, u'NWriteBytes': 98130390, u'NIndexBlockWrites': 11418, u'NBytesIncoming': 18091440, u'NSyncs': 9174, u'NInserts': 123287, u'NWriteIOs': 20592}}}}} 2021-12-15 21:19:08,062 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.2874740660190582, 0.25409772992134094, 0.2755286693572998, 0.2202894389629364, 0.3110775649547577, 0.32433322072029114, 0.2815815806388855, 0.3106657862663269, 0.316855251789093, 0.3280799388885498, 0.33990737795829773, 0.35560673475265503, 0.34935957193374634, 0.3439358174800873, 0.34455549716949463, 0.35352957248687744, 0.3226039707660675, 0.3061531186103821, 0.36623305082321167, 0.34518617391586304, 0.28205451369285583, 0.3270479440689087, 0.3102247714996338, 0.3487279713153839, 0.3215615749359131, 0.327037513256073, 0.32316669821739197, 0.28674808144569397, 0.2619151473045349, 0.33298423886299133, 0.3006667196750641, 0.31760090589523315, 0.34969931840896606, 0.2608032822608948, 0.3126251995563507, 0.3084346652030945, 0.2829858660697937, 0.2915838956832886, 0.25966107845306396, 0.275647908449173, 0.2848673462867737, 0.2533264458179474, 0.2755967080593109, 0.2737073600292206, 0.24214854836463928, 0.23755893111228943, 0.21767470240592957, 0.27799734473228455, 0.22025662660598755, 0.2251298725605011, 0.26071199774742126, 0.2697267234325409, 0.23199117183685303, 0.2280936986207962, 0.2751094102859497, 0.28072816133499146, 0.30473339557647705, 0.2826727628707886, 0.25582221150398254, 0.3138326108455658, 0.2713284194469452, 0.28083160519599915, 0.32834428548812866, 0.22685179114341736], '172.23.110.66': [0.29303786158561707, 0.27243316173553467, 0.2636096775531769, 0.33267804980278015, 0.3075178861618042, 0.32344868779182434, 0.33151665329933167, 0.33656999468803406, 0.3467898964881897, 0.3039814531803131, 0.35273829102516174, 0.2925255000591278, 0.376753032207489, 0.2423972636461258, 0.3365459144115448, 0.3277125954627991, 0.33694496750831604, 0.33749714493751526, 0.32414910197257996, 0.3504079282283783, 0.31578001379966736, 0.34735631942749023, 0.32527023553848267, 0.33649882674217224, 0.34905263781547546, 0.3310573399066925, 0.32932522892951965, 0.3545362055301666, 0.36005645990371704, 0.36263832449913025, 0.37528321146965027, 0.36265450716018677, 0.37165698409080505, 0.32370200753211975, 0.3851180970668793, 0.358816534280777, 0.3563327491283417, 0.3134361207485199, 0.36983466148376465, 0.3716394603252411, 0.3798007667064667, 0.3095687925815582, 0.30518513917922974, 0.37820130586624146, 0.2988226115703583, 0.3573448359966278, 0.3451266586780548, 0.3720130920410156, 0.3553673326969147, 0.37569600343704224, 0.3660520911216736, 0.35414978861808777, 0.3620513081550598, 0.3728456199169159, 0.35059916973114014, 0.3648075461387634, 0.3457244634628296, 0.3574279844760895, 0.2924576997756958, 0.3423195481300354, 0.3744142949581146, 0.3532250225543976, 0.37527769804000854, 0.3074757754802704], '172.23.110.64': [0.32849225401878357, 0.35308781266212463, 0.3808501362800598, 0.31886136531829834, 0.34424325823783875, 0.3168197572231293, 0.33927619457244873, 0.308817058801651, 0.34358900785446167, 0.30293747782707214, 0.2963379919528961, 0.33474427461624146, 0.30281901359558105, 0.29963284730911255, 0.30697765946388245, 0.3170049488544464, 0.3478377163410187, 0.3150741755962372, 0.25116851925849915, 0.34344276785850525, 0.35303881764411926, 0.3066275417804718, 0.30694690346717834, 0.2632383704185486, 0.29523515701293945, 0.2676019072532654, 0.30274972319602966, 0.32237350940704346, 0.32576784491539, 0.3152596056461334, 0.3337748944759369, 0.2919376492500305, 0.29190686345100403, 0.33545324206352234, 0.30960479378700256, 0.3176226317882538, 0.3706207573413849, 0.309396892786026, 0.37125924229621887, 0.3499552011489868, 0.347943514585495, 0.3563360571861267, 0.31823301315307617, 0.33508703112602234, 0.3274742364883423, 0.2791077494621277, 0.26838240027427673, 0.34321755170822144, 0.3345223069190979, 0.3231208920478821, 0.3521534502506256, 0.3613525331020355, 0.3738606870174408, 0.3650074899196625, 0.38171008229255676, 0.2642926275730133, 0.36779361963272095, 0.30372101068496704, 0.32945847511291504, 0.36415407061576843, 0.3285408914089203, 0.3269526958465576, 0.3684670627117157, 0.360507607460022]} 2021-12-15 21:19:08,265 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 33.0 2021-12-15 21:19:08,539 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 33.0 2021-12-15 21:19:08,756 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 33.0 2021-12-15 21:19:08,756 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 33.0, '172.23.110.66': 33.0, '172.23.110.64': 33.0} ############################################################ # # Step 7: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 5004.16015625 # ############################################################ 2021-12-15 21:19:08,799 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:19:08,799 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-15 21:20:14,805 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:20:14,805 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:20:19,809 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:20:19,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:20,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:20:20,167 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:20,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:20:20,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:20:20,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:20:20,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:20:20,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:20:20,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:20:20,239 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:20:20,252 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:20:20,252 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:20:20,305 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['108264 memcached\n'], []) 2021-12-15 21:20:20,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:20:20,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:20:20,392 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['108264 memcached\n'], []) 2021-12-15 21:20:20,392 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:20:20,395 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:20:20,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:20,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:20:20,839 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:20,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:20:20,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:20:20,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:20:20,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:20:20,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:20:20,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:20:20,926 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:20:20,940 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:20:20,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:20:21,000 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['67962 memcached\n'], []) 2021-12-15 21:20:21,002 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:20:21,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:20:21,122 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['67962 memcached\n'], []) 2021-12-15 21:20:21,124 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:20:21,125 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:20:21,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:22,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:20:22,493 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:22,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:20:22,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:20:22,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:20:22,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:20:22,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:20:22,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:20:22,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:20:22,644 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:20:22,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:20:22,721 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['170597 memcached\n'], []) 2021-12-15 21:20:22,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:20:22,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:20:22,868 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['170597 memcached\n'], []) 2021-12-15 21:20:22,869 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:20:22,871 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:20:22,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:23,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:20:23,306 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:23,306 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:20:23,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:20:23,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:20:23,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:20:23,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:20:23,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:20:23,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:20:23,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:20:23,454 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:20:23,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:20:23,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:23,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:23,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:23,665 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:23,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:23,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:23,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:23,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:23,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:23,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:23,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:24,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:24,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:24,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:24,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,234 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:24,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:24,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:24,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:24,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:24,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:24,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:24,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:24,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:20:24,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:20:24,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:24,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:24,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:20:24,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:20:24,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:25,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:25,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:20:25,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:20:25,117 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:25,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:25,188 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:20:25,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:20:25,237 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:20:25,256 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:20:25,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:20:25,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:25,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:20:25,312 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:20:25,313 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:20:25,315 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:20:25,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:28,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:20:28,329 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:28,331 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:20:28,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:20:28,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:20:28,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:20:28,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:20:28,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:20:28,436 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:20:28,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:20:28,469 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:20:28,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:20:28,517 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:28,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:28,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:28,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:28,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:28,729 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:28,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:28,811 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:28,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:28,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:28,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:28,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:28,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:28,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:29,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:29,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,063 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:29,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:29,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,150 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:29,165 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:29,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:29,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:29,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,267 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:29,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:29,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:20:29,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:20:29,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:29,426 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:20:29,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:20:29,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:20:29,459 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:20:29,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:20:29,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:29,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:20:29,496 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:20:29,496 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:20:29,499 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:20:29,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:29,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:20:30,013 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:30,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:20:30,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:20:30,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:20:30,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:20:30,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:20:30,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:20:30,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:20:30,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:20:30,160 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:20:30,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:20:30,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:30,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:30,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:30,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:20:30,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:30,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:30,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:30,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:20:30,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:30,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:30,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:30,676 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:20:30,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:30,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:30,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:30,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:20:30,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:30,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:30,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:30,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:20:31,005 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:31,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:31,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:31,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:20:31,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:31,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:31,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:31,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:20:31,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:31,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:20:31,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:31,357 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:20:31,372 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:20:31,387 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:20:31,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:20:31,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:20:31,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:20:31,427 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:20:31,427 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:20:31,430 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:20:31,433 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:20:31,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:20:32,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:20:32,905 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:20:32,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:20:32,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:20:32,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:20:32,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:20:32,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:20:33,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:20:33,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:20:33,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:33,198 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:33,207 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:33,207 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:35,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:35,427 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:35,434 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:35,434 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:37,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:37,604 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:37,609 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:37,609 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:39,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:39,773 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:39,778 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:39,779 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:41,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:41,946 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:41,951 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:41,951 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:43,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:44,134 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:44,140 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:44,141 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:46,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:46,253 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:46,259 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:46,259 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:48,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:48,437 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:48,443 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:48,443 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:50,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:50,611 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:50,618 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:50,618 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:52,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:52,780 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:52,785 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:52,785 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:54,785 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:54,917 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:54,921 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:54,921 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:56,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:57,085 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:57,089 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:57,089 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:20:59,089 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:20:59,220 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:20:59,223 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:20:59,223 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:01,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:01,404 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:21:01,408 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:01,408 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:03,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:03,591 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:21:03,595 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:03,595 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:05,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:05,756 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:21:05,760 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:05,760 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:07,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:07,944 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:21:07,950 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:07,950 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:09,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:10,118 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:10,118 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:12,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:12,283 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:12,283 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:14,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:14,436 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:14,436 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:16,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:16,584 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:16,584 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:18,584 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:18,743 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:18,743 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:20,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:20,894 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:20,894 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:22,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:23,042 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:23,043 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:25,043 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:25,190 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:25,190 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:27,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:27,355 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:27,355 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:29,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:29,586 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:29,588 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:31,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:31,743 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:31,743 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:33,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:33,911 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:33,913 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:35,913 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:36,084 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:36,084 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:38,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:38,247 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:38,247 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:40,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:40,414 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:40,414 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:42,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:42,566 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:42,566 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:44,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:44,729 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:44,729 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:46,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:46,885 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:46,885 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:48,887 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:49,048 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:49,049 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:51,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:51,214 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:51,216 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:53,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:53,382 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:53,382 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:55,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:55,530 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:55,530 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:57,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:57,687 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:57,687 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:21:59,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:21:59,855 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:21:59,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:01,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:02,019 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:02,019 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:04,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:04,194 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:04,194 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:06,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:06,441 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:06,441 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:08,443 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:08,750 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:08,750 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:10,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:11,789 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:11,789 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:13,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:14,153 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:14,153 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:16,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:16,523 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:16,523 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:18,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:18,970 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:18,970 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:20,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:21,502 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:21,502 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:22:23,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:24,266 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:22:24,266 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 112.836999893. 2021-12-15 21:22:24,266 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:22:24,269 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:22:24,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:22:24,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:22:25,302 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:22:25,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:22:25,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:22:25,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:22:25,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:22:25,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:22:25,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:22:25,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:22:25,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:25,674 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:25,674 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:22:27,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:27,917 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:27,917 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:22:29,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:30,608 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:30,608 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:22:32,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:32,904 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:32,905 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:22:34,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:39,651 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:22:39,651 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 15.3849999905. 2021-12-15 21:22:39,653 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:22:39,654 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:22:39,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:22:40,545 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:22:40,627 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:22:40,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:22:40,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:22:40,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:22:40,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:22:40,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:22:40,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:22:40,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:22:41,176 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:41,418 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:41,418 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:22:43,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:43,591 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:43,591 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:22:45,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:46,055 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:22:46,055 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:22:48,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:22:48,285 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:22:48,285 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 8.63199996948. ############################################################ # # RAM FootPrint: 5037.1015625 # ############################################################ 2021-12-15 21:22:48,319 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:22:48,319 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 85 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-15 21:24:13,323 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:24:13,323 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:24:18,325 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:24:18,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:19,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:24:19,608 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:19,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:24:19,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:24:19,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:24:19,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:24:19,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:24:19,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:24:19,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:24:19,871 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:24:19,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:24:19,961 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['130205 memcached\n'], []) 2021-12-15 21:24:19,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:24:20,112 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:24:20,262 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['130205 memcached\n'], []) 2021-12-15 21:24:20,262 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:24:20,265 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:24:20,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:21,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:24:21,154 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:21,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:24:21,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:24:21,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:24:21,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:24:21,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:24:21,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:24:21,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:24:21,375 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:24:21,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:24:21,460 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['89242 memcached\n'], []) 2021-12-15 21:24:21,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:24:21,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:24:21,767 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['89242 memcached\n'], []) 2021-12-15 21:24:21,769 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:24:21,772 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:24:21,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:22,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:24:22,312 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:22,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:24:22,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:24:22,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:24:22,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:24:22,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:24:22,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:24:22,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:24:22,497 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:24:22,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:24:22,588 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['176081 memcached\n'], []) 2021-12-15 21:24:22,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:24:22,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:24:22,746 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['176081 memcached\n'], []) 2021-12-15 21:24:22,746 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:24:22,750 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:24:22,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:23,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:24:23,194 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:23,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:24:23,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:24:23,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:24:23,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:24:23,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:24:23,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:24:23,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:24:23,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:24:23,344 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:24:23,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:24:23,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:23,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:23,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:23,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:23,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:23,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:23,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:23,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:23,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:23,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:23,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:23,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:23,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:23,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:23,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:23,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:24,003 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,032 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:24,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:24,105 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:24,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:24,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:24,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:24,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:24,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:24,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:24:24,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:24:24,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:24:24,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:24:24,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,653 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:24:24,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:24:24,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,770 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:24,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:24:24,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:24:24,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:24:24,864 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:24:24,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:24:24,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:24,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:24:24,911 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:24:24,911 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:24:24,913 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:24:25,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:25,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:24:25,427 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:25,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:24:25,446 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:24:25,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:24:25,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:24:25,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:24:25,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:24:25,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:24:25,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:24:25,559 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:24:25,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:24:25,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:25,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:25,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:25,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:25,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:25,818 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:25,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:25,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:25,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:25,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:26,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:26,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:26,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:26,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:26,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:26,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:26,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:26,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:26,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:26,492 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:26,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:26,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,635 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:24:26,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:24:26,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:26,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:24:26,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:24:26,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:24:26,890 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:24:26,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:24:26,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:26,926 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:24:26,941 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:24:26,941 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:24:26,944 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:24:27,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:27,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:24:27,441 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:27,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:24:27,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:24:27,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:24:27,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:24:27,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:24:27,530 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:24:27,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:24:27,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:24:27,578 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:24:27,604 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:24:27,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:27,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:27,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:27,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:24:27,780 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:27,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:27,846 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:27,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:24:27,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:27,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:27,982 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:28,007 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:24:28,032 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:28,086 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:28,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:24:28,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:28,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:28,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:24:28,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:28,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:28,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:24:28,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:28,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:28,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:24:28,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:24:28,479 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:28,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:24:28,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:24:28,525 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:24:28,525 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:24:28,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:24:28,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:24:28,563 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:24:28,563 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:24:28,563 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:24:28,565 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:24:28,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:24:28,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:24:29,061 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:24:29,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:24:29,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:24:29,096 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:24:29,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:24:29,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:24:29,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:24:29,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:24:29,186 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:31,131 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:24:31,137 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:31,137 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:33,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:33,303 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:33,308 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:33,308 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:35,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:35,486 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:35,490 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:35,490 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:37,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:37,596 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:37,601 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:37,601 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:39,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:39,782 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:39,789 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:39,789 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:41,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:41,960 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:41,963 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:41,963 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:43,964 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:44,147 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:44,154 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:44,154 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:46,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:46,322 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:46,326 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:46,326 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:48,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:48,515 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:48,519 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:48,519 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:50,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:50,701 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:50,707 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:50,707 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:52,709 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:52,867 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:24:52,871 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:52,871 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:54,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:55,038 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:55,038 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:57,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:57,181 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:57,181 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:24:59,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:24:59,331 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:24:59,331 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:01,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:01,482 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:01,482 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:03,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:03,671 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:03,671 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:05,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:05,823 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:05,825 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:07,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:07,970 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:07,971 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:09,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:10,114 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:10,114 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:12,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:12,276 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:12,278 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:14,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:14,473 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:14,473 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:16,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:16,651 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:16,653 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:18,653 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:18,803 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:18,805 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:20,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:20,966 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:20,966 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:22,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:23,128 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:23,128 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:25,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:25,295 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:25,296 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:27,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:27,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:27,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:29,451 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:29,598 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:29,598 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:31,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:31,750 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:31,750 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:33,752 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:33,915 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:33,915 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:35,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:36,076 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:36,076 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:38,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:38,243 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:38,243 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:40,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:40,403 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:40,403 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:42,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:42,558 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:42,558 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:44,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:44,720 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:44,720 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:46,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:46,875 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:46,875 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:48,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:49,049 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:49,049 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:51,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:51,338 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:51,338 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:53,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:53,562 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:53,563 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:55,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:55,783 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:55,783 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:57,785 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:25:57,990 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:25:57,990 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:25:59,992 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:00,160 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:00,160 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:02,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:02,322 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:02,322 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:04,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:04,503 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:04,503 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:06,505 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:06,667 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:06,667 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:08,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:08,943 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:08,944 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:10,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:11,214 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:11,216 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:26:13,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:15,342 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:26:15,342 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 106.779000044. 2021-12-15 21:26:15,344 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:26:15,345 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:26:15,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:26:15,719 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:26:15,796 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:26:15,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:26:15,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:26:15,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:26:15,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:26:15,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:26:15,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:26:15,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:26:15,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:16,194 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:26:16,194 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:26:18,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:18,394 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:26:18,394 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.04999995232. 2021-12-15 21:26:18,395 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:26:18,397 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:26:19,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:26:19,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:26:20,285 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:26:20,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:26:20,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:26:20,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:26:20,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:26:20,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:26:20,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:26:27,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:26:27,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:26:28,098 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:26:28,098 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 9.70300006866. ############################################################ # # RAM FootPrint: 5018.49609375 # ############################################################ 2021-12-15 21:26:28,128 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:26:28,128 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 119 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-15 21:28:27,131 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:28:27,131 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:28:32,132 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:28:32,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:32,737 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:28:32,826 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:32,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:28:32,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:28:32,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:28:32,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:28:32,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:28:32,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:28:32,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:28:32,950 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:28:32,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:28:33,017 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['131945 memcached\n'], []) 2021-12-15 21:28:33,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:28:33,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:28:33,118 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['131945 memcached\n'], []) 2021-12-15 21:28:33,118 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:28:33,121 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:28:33,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:37,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:28:37,492 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:37,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:28:37,505 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:28:37,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:28:37,530 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:28:37,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:28:37,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:28:37,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:28:37,578 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:28:37,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:28:37,628 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['90290 memcached\n'], []) 2021-12-15 21:28:37,630 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:28:37,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:28:37,756 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['90290 memcached\n'], []) 2021-12-15 21:28:37,756 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:28:37,759 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:28:38,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:38,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:28:38,365 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:38,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:28:38,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:28:38,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:28:38,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:28:38,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:28:38,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:28:38,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:28:38,542 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:28:38,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:28:38,628 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['177437 memcached\n'], []) 2021-12-15 21:28:38,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:28:38,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:28:38,772 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['177437 memcached\n'], []) 2021-12-15 21:28:38,772 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:28:38,775 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:28:38,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:39,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:28:39,211 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:39,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:28:39,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:28:39,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:28:39,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:28:39,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:28:39,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:28:39,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:28:39,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:28:39,377 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:28:39,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:28:39,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:39,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:39,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:39,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:39,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:39,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:39,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:39,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:39,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:39,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:39,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:39,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:39,913 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:39,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:39,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:40,012 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:40,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:40,150 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:40,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:40,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:40,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,426 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:40,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:40,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:40,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:40,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:28:40,770 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:28:40,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:40,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:40,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:28:40,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:28:40,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:41,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:41,066 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:28:41,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:28:41,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:41,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:41,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:28:41,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:28:41,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:41,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:41,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:28:41,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:28:41,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:28:41,398 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:28:41,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:28:41,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:41,433 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:28:41,447 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:28:41,447 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:28:41,450 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:28:41,573 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:41,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:28:41,963 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:41,964 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:28:41,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:28:42,002 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:28:42,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:28:42,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:28:42,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:28:42,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:28:42,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:28:42,107 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:28:42,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:28:42,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:42,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:42,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:42,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:42,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:42,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:42,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:42,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:42,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:42,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:42,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:42,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:42,926 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:42,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:42,989 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:43,028 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:43,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:43,081 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:43,112 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:43,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:43,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:43,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:43,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:43,267 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:28:43,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:28:43,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:43,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:43,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:28:43,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:28:43,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:28:43,457 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:28:43,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:28:43,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:43,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:28:43,503 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:28:43,503 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:28:43,505 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:28:43,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:43,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:28:44,013 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:44,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:28:44,028 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:28:44,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:28:44,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:28:44,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:28:44,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:28:44,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:28:44,112 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:28:44,125 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:28:44,150 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:28:44,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:44,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:28:44,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,331 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:44,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:28:44,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:44,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:28:44,553 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:44,647 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:28:44,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:44,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:28:44,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:44,849 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:28:44,867 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:44,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:44,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:28:44,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:44,982 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:28:45,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:45,025 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:28:45,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:28:45,049 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:28:45,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:28:45,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:28:45,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:28:45,084 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:28:45,084 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:28:45,084 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:28:45,085 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:28:45,188 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:28:45,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:28:45,572 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:28:45,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:28:45,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:28:45,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:28:45,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:28:45,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:28:45,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:28:45,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:28:45,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:45,885 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:45,890 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:45,890 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:47,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:48,072 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:48,075 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:48,075 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:50,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:50,260 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:50,265 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:50,265 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:52,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:52,446 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:52,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:52,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:54,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:54,614 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:54,618 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:54,618 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:56,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:56,720 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:56,724 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:56,724 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:28:58,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:28:58,877 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:28:58,881 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:28:58,881 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:00,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:01,015 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:29:01,020 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:01,022 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:03,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:03,188 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:29:03,193 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:03,193 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:05,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:05,403 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:05,403 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:07,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:07,552 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:07,552 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:09,553 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:09,701 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:09,701 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:11,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:11,858 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:11,858 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:13,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:14,009 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:14,009 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:16,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:16,206 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:16,207 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:18,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:18,349 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:18,349 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:20,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:20,506 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:20,506 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:22,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:22,697 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:22,698 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:24,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:24,941 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:24,943 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:26,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:27,101 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:27,101 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:29,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:29,273 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:29,273 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:31,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:31,440 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:31,440 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:33,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:33,592 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:33,592 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:35,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:35,759 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:35,759 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:37,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:37,921 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:37,921 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:39,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:40,075 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:40,075 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:42,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:42,239 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:42,239 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:44,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:44,405 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:44,405 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:46,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:46,566 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:46,566 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:48,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:48,729 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:48,729 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:50,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:50,895 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:50,895 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:52,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:53,053 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:53,053 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:55,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:55,214 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:55,214 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:57,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:57,420 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:57,420 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:29:59,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:29:59,621 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:29:59,621 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:01,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:01,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:01,801 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:03,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:03,976 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:03,976 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:05,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:06,153 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:06,153 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:08,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:08,312 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:08,312 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:10,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:10,479 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:10,479 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:12,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:12,654 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:12,654 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:14,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:14,828 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:14,828 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:16,829 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:17,002 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:17,003 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:19,003 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:19,165 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:19,167 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:30:21,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:21,563 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:30:21,563 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.4799997807. 2021-12-15 21:30:21,565 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:30:21,566 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:30:21,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:30:21,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:30:21,976 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:30:21,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:30:22,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:30:22,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:30:22,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:30:22,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:30:22,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:30:22,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:30:22,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:22,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:22,349 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:24,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:24,536 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:24,536 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:26,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:26,726 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:26,726 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:28,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:28,923 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:28,923 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:30,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:31,098 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:31,098 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:33,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:33,279 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:30:33,279 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:30:35,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:41,618 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:30:41,619 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 20.0559999943. 2021-12-15 21:30:41,621 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:30:41,624 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:30:41,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:30:42,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:30:43,092 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:30:43,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:30:43,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:30:43,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:30:43,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:30:43,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:30:43,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:30:43,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:30:43,339 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:30:43,535 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:30:43,536 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.91499996185. ############################################################ # # RAM FootPrint: 4987.1015625 # ############################################################ 2021-12-15 21:30:43,566 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:30:43,566 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 95 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-15 21:32:18,569 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:32:18,571 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:32:23,572 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:32:23,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:24,163 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:32:24,437 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:24,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:32:24,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:32:24,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:32:24,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:32:24,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:32:24,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:32:24,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:32:24,720 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:32:24,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:32:24,826 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['133481 memcached\n'], []) 2021-12-15 21:32:24,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:32:24,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:32:25,007 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['133481 memcached\n'], []) 2021-12-15 21:32:25,009 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:32:25,010 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:32:25,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:25,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:32:25,645 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:25,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:32:25,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:32:25,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:32:25,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:32:25,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:32:25,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:32:25,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:32:25,920 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:32:25,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:32:26,072 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['91313 memcached\n'], []) 2021-12-15 21:32:26,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:32:26,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:32:26,252 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['91313 memcached\n'], []) 2021-12-15 21:32:26,252 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:32:26,255 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:32:26,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:26,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:32:26,864 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:26,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:32:26,887 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:32:26,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:32:26,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:32:26,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:32:26,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:32:27,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:32:27,029 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:32:27,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:32:27,121 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['178757 memcached\n'], []) 2021-12-15 21:32:27,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:32:27,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:32:27,256 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['178757 memcached\n'], []) 2021-12-15 21:32:27,256 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:32:27,259 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:32:27,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:27,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:32:27,710 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:27,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:32:27,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:32:27,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:32:27,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:32:27,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:32:27,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:32:27,834 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:32:27,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:32:27,871 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:32:27,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:32:27,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:27,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:28,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:28,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:28,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:28,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:28,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:28,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:28,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:28,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:28,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:28,614 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:28,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:28,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,734 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:28,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:28,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:28,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:28,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:28,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:28,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,007 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:29,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:29,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:29,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:29,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:29,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:29,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,584 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:32:29,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:32:29,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:32:29,770 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:32:29,813 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:29,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:29,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:32:29,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:32:29,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:30,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:30,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:32:30,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:32:30,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:32:30,111 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:32:30,112 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:32:30,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:30,150 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:32:30,165 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:32:30,167 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:32:30,168 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:32:30,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:30,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:32:30,684 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:30,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:32:30,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:32:30,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:32:30,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:32:30,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:32:30,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:32:30,795 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:32:30,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:32:30,829 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:32:30,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:32:30,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:30,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:30,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:31,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:31,043 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,089 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:31,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:31,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:31,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:31,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:31,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:31,471 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:31,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:31,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:31,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:31,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:31,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:31,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:31,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:31,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:31,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:31,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:31,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:32,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:32,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:32,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:32:32,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:32:32,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:32,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:32,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:32:32,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:32:32,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:32:32,209 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:32:32,209 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:32:32,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:32,239 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:32:32,252 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:32:32,252 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:32:32,253 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:32:32,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:32,704 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:32:32,780 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:32,780 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:32:32,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:32:32,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:32:32,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:32:32,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:32:32,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:32:32,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:32:32,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:32:32,917 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:32:32,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:32:32,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:33,068 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:32:33,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:33,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:32:33,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,288 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:33,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:32:33,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,443 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:33,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:32:33,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:33,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:32:33,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:33,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:32:33,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:33,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:32:33,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:33,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:32:33,811 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:32:33,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:33,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:32:33,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:32:33,867 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:32:33,867 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:32:33,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:32:33,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:32:33,911 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:32:33,911 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:32:33,911 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:32:33,914 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:32:34,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:32:34,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:32:34,513 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:32:34,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:32:34,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:32:34,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:32:34,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:32:34,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:32:34,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:32:34,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:32:34,619 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:34,994 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:32:34,999 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:34,999 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:36,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:37,155 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:37,160 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:37,160 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:39,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:39,328 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:39,335 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:39,335 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:41,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:41,506 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:41,512 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:41,512 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:43,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:43,697 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:43,703 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:43,703 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:45,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:45,875 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:45,878 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:45,880 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:47,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:48,068 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:48,072 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:48,072 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:50,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:50,240 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:50,246 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:50,246 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:52,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:52,420 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:52,424 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:52,424 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:54,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:54,579 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:54,585 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:54,585 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:56,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:56,749 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:32:56,753 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:56,753 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:32:58,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:32:58,894 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:32:58,894 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:00,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:01,039 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:01,039 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:03,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:03,191 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:03,191 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:05,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:05,342 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:05,344 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:07,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:07,494 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:07,494 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:09,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:09,641 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:09,642 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:11,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:11,788 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:11,789 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:13,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:13,950 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:13,950 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:15,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:16,167 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:16,167 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:18,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:18,398 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:18,400 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:20,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:20,561 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:20,561 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:22,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:22,724 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:22,724 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:24,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:24,884 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:24,884 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:26,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:27,048 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:27,048 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:29,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:29,209 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:29,209 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:31,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:31,368 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:31,368 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:33,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:33,526 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:33,526 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:35,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:35,688 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:35,690 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:37,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:37,857 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:37,857 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:39,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:40,022 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:40,022 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:42,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:42,180 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:42,180 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:44,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:44,345 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:44,346 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:46,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:46,494 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:46,496 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:48,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:48,851 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:48,851 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:50,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:51,025 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:51,025 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:53,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:53,226 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:53,226 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:55,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:55,403 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:55,403 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:57,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:57,576 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:57,578 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:33:59,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:33:59,742 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:33:59,742 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:01,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:01,907 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:01,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:03,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:04,082 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:04,082 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:06,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:06,253 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:06,253 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:08,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:08,421 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:08,421 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:10,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:10,595 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:10,595 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:34:12,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:12,763 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:34:12,763 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.8510000706. 2021-12-15 21:34:12,765 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:34:12,766 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:34:12,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:34:13,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:34:13,221 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:34:13,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:34:13,262 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:34:13,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:34:13,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:34:13,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:34:13,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:34:13,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:34:13,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:13,586 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:13,588 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:34:15,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:15,766 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:15,767 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:34:17,767 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:17,944 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:34:17,944 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:34:19,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:20,135 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:34:20,137 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.37199997902. 2021-12-15 21:34:20,137 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:34:20,138 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:34:20,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:34:20,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:34:21,069 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:34:21,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:34:21,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:34:21,109 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:34:21,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:34:21,165 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:34:21,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:34:21,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:34:21,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:34:21,361 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:34:21,361 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.22499990463. ############################################################ # # RAM FootPrint: 4962.57421875 # ############################################################ 2021-12-15 21:34:21,391 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:34:21,391 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 64 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-15 21:35:25,395 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:35:25,397 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:35:30,398 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:35:30,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:30,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:35:31,101 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:31,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:35:31,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:35:31,334 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:35:31,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:35:31,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:35:31,510 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:35:31,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:35:31,562 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:35:31,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:35:31,645 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['135159 memcached\n'], []) 2021-12-15 21:35:31,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:35:31,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:35:31,772 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['135159 memcached\n'], []) 2021-12-15 21:35:31,772 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:35:31,775 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:35:33,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:34,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:35:34,263 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:34,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:35:34,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:35:34,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:35:34,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:35:34,336 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:35:34,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:35:34,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:35:34,371 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:35:34,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:35:34,420 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['92393 memcached\n'], []) 2021-12-15 21:35:34,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:35:34,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:35:34,651 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['92393 memcached\n'], []) 2021-12-15 21:35:34,651 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:35:34,653 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:35:34,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:35,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:35:35,157 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:35,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:35:35,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:35:35,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:35:35,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:35:35,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:35:35,270 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:35:35,288 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:35:35,308 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:35:35,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:35:35,391 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['180112 memcached\n'], []) 2021-12-15 21:35:35,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:35:35,471 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:35:35,548 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['180112 memcached\n'], []) 2021-12-15 21:35:35,548 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:35:35,549 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:35:35,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:35,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:35:35,953 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:35,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:35:35,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:35:35,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:35:36,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:35:36,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:35:36,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:35:36,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:35:36,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:35:36,109 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:35:36,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:35:36,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:36,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:36,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:36,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:36,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:36,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:36,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:36,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:36,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:36,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:36,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:36,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:36,982 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:37,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:37,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:37,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:37,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:37,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:37,275 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:37,362 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:37,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,451 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:35:37,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:35:37,492 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:35:37,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:35:37,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:35:37,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:35:37,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:37,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:35:37,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:35:37,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:35:37,756 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:35:37,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:35:37,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:37,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:35:37,796 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:35:37,796 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:35:37,799 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:35:37,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:38,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:35:38,319 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:38,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:35:38,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:35:38,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:35:38,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:35:38,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:35:38,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:35:38,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:35:38,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:35:38,463 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:35:38,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:35:38,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:38,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:38,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:38,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:38,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:38,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:38,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:38,788 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:38,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:38,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:38,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:38,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:38,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:38,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,033 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:39,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:39,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:39,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:39,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:39,275 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:39,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:39,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:39,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:39,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:39,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:39,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:39,686 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:35:39,790 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:35:39,816 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:39,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:35:39,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:35:39,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:35:39,924 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:35:39,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:35:39,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:39,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:35:39,970 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:35:39,970 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:35:39,973 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:35:40,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:40,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:35:40,493 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:40,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:35:40,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:35:40,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:35:40,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:35:40,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:35:40,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:35:40,584 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:35:40,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:35:40,611 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:35:40,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:35:40,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:40,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:40,755 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:40,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:35:40,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:40,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:40,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:40,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:35:40,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,048 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:41,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:35:41,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:41,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:35:41,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,219 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:41,262 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:35:41,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:41,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:35:41,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:41,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:35:41,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:41,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:35:41,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:35:41,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:41,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:35:41,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:35:41,539 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:35:41,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:35:41,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:35:41,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:35:41,571 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:35:41,571 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:35:41,571 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:35:41,573 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:35:41,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:35:42,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:35:42,450 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:35:42,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:35:42,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:35:42,492 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:35:42,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:35:42,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:35:42,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:35:42,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:35:42,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:42,727 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:42,733 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:42,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:44,734 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:44,838 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:44,842 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:44,842 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:46,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:47,003 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:47,009 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:47,009 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:49,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:49,173 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:49,176 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:49,177 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:51,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:51,349 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:51,355 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:51,355 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:53,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:53,533 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:53,538 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:53,539 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:55,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:55,707 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:55,710 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:55,710 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:57,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:35:57,861 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:35:57,865 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:35:57,865 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:35:59,867 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:00,049 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:36:00,052 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:00,052 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:02,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:02,190 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:36:02,194 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:02,194 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:04,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:04,345 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:04,345 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:06,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:06,500 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:06,500 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:08,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:08,644 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:08,644 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:10,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:10,799 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:10,799 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:12,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:12,947 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:12,947 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:14,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:15,105 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:15,105 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:17,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:17,250 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:17,250 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:19,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:19,398 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:19,398 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:21,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:21,621 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:21,621 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:23,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:23,854 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:23,854 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:25,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:26,015 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:26,015 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:28,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:28,170 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:28,170 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:30,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:30,328 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:30,328 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:32,329 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:32,489 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:32,489 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:34,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:34,648 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:34,648 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:36,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:36,799 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:36,799 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:38,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:38,961 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:38,961 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:40,961 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:41,122 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:41,122 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:43,122 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:43,286 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:43,286 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:45,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:45,447 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:45,447 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:47,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:47,598 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:47,598 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:49,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:49,749 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:49,749 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:51,749 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:51,905 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:51,905 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:53,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:54,065 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:54,065 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:56,066 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:56,516 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:56,516 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:36:58,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:36:58,691 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:36:58,693 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:00,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:00,888 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:00,888 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:02,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:03,052 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:03,052 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:05,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:05,223 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:05,223 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:07,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:07,405 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:07,405 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:09,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:09,573 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:09,575 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:11,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:11,756 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:11,756 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:13,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:13,927 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:13,928 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:15,928 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:16,095 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:16,095 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:18,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:18,266 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:18,266 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:37:20,267 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:20,539 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:37:20,539 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.9689998627. 2021-12-15 21:37:20,542 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:37:20,542 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:37:20,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:37:20,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:37:21,089 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:37:21,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:37:21,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:37:21,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:37:21,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:37:21,163 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:37:21,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:37:21,242 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:37:21,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:21,430 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:21,430 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:37:23,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:23,601 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:23,601 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:37:25,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:25,756 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:25,756 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:37:27,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:27,930 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:37:27,930 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:37:29,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:32,335 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:37:32,335 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.7940001488. 2021-12-15 21:37:32,335 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:37:32,338 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:37:32,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:37:33,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:37:33,259 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:37:33,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:37:33,295 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:37:33,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:37:33,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:37:33,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:37:33,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:37:33,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:37:33,505 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:37:33,704 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:37:33,704 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.36899995804. ############################################################ # # RAM FootPrint: 4949.6796875 # ############################################################ 2021-12-15 21:37:33,733 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:37:33,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 87 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-15 21:39:00,737 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:39:00,737 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:39:05,740 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:39:05,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:06,775 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:39:06,894 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:06,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:39:06,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:39:06,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:39:06,969 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:39:06,987 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:39:07,033 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:39:07,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:39:07,082 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:39:07,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:39:07,138 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['136648 memcached\n'], []) 2021-12-15 21:39:07,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:39:07,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:39:07,283 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['136648 memcached\n'], []) 2021-12-15 21:39:07,283 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:39:07,286 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:39:07,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:08,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:39:08,859 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:08,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:39:08,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:39:08,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:39:08,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:39:08,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:39:09,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:39:09,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:39:09,178 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:39:09,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:39:09,288 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['93379 memcached\n'], []) 2021-12-15 21:39:09,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:39:09,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:39:09,519 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['93379 memcached\n'], []) 2021-12-15 21:39:09,519 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:39:09,522 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:39:09,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:09,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:39:10,075 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:10,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:39:10,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:39:10,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:39:10,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:39:10,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:39:10,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:39:10,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:39:10,244 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:39:10,244 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:39:10,321 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['181407 memcached\n'], []) 2021-12-15 21:39:10,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:39:10,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:39:10,480 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['181407 memcached\n'], []) 2021-12-15 21:39:10,480 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:39:10,483 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:39:10,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:10,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:39:10,911 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:10,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:39:10,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:39:10,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:39:10,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:39:10,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:39:11,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:39:11,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:39:11,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:39:11,055 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:39:11,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:39:11,104 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:11,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:11,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:11,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:11,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:11,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:11,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:11,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:11,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:11,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:11,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:11,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:11,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:11,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:11,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:12,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:12,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:12,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:12,206 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:12,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:12,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:39:12,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:39:12,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,573 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:39:12,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:39:12,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:39:12,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:39:12,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:12,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:39:12,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:39:12,877 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:39:12,895 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:39:12,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:39:12,913 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:12,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:39:12,948 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:39:12,948 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:39:12,951 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:39:13,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:13,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:39:13,437 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:13,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:39:13,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:39:13,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:39:13,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:39:13,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:39:13,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:39:13,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:39:13,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:39:13,578 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:39:13,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:39:13,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:13,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:13,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:13,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:13,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:13,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:13,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:13,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:13,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:13,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:14,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:14,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,142 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:14,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:14,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:14,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:14,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:14,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:14,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,446 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:14,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:14,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:14,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:14,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:14,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:14,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:39:14,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:39:14,813 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:14,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:39:14,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:39:14,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:39:14,888 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:39:14,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:39:14,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:14,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:39:14,921 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:39:14,921 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:39:14,924 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:39:15,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:15,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:39:15,461 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:15,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:39:15,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:39:15,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:39:15,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:39:15,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:39:15,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:39:15,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:39:15,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:39:15,591 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:39:15,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:39:15,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:15,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:15,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:15,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:39:15,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:15,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:15,834 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:15,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:39:15,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:15,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:15,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:15,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:39:15,969 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:15,989 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:16,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:39:16,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:16,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:39:16,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,142 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:16,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:39:16,170 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:16,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:39:16,242 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,262 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:16,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:39:16,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:39:16,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:16,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:39:16,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:39:16,371 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:39:16,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:39:16,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:39:16,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:39:16,401 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:39:16,401 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:39:16,401 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:39:16,404 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:39:16,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:39:17,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:39:18,030 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:39:18,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:39:18,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:39:18,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:39:18,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:39:18,109 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:39:18,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:39:18,142 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:39:18,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:18,302 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:18,309 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:18,309 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:20,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:20,474 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:20,479 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:20,479 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:22,479 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:22,645 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:22,651 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:22,651 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:24,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:24,767 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:24,773 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:24,773 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:26,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:26,936 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:26,940 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:26,940 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:28,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:29,109 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:29,114 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:29,114 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:31,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:31,282 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:31,286 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:31,286 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:33,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:33,460 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:39:33,463 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:33,463 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:35,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:35,611 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:35,611 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:37,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:37,759 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:37,760 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:39,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:39,930 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:39,931 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:41,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:42,079 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:42,079 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:44,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:44,223 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:44,223 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:46,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:46,369 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:46,369 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:48,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:48,519 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:48,519 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:50,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:50,674 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:50,674 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:52,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:52,905 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:52,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:54,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:55,072 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:55,072 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:57,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:57,227 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:57,227 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:39:59,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:39:59,388 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:39:59,388 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:01,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:01,543 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:01,543 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:03,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:03,697 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:03,697 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:05,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:05,851 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:05,851 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:07,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:08,007 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:08,009 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:10,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:10,163 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:10,164 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:12,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:12,322 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:12,322 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:14,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:14,493 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:14,493 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:16,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:16,658 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:16,658 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:18,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:18,838 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:18,838 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:20,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:20,993 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:20,993 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:22,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:23,155 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:23,155 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:25,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:25,313 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:25,313 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:27,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:27,474 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:27,474 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:29,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:29,621 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:29,622 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:31,622 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:31,779 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:31,779 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:33,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:33,938 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:33,940 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:35,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:36,101 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:36,102 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:38,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:38,270 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:38,270 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:40,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:41,158 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:41,158 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:43,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:43,331 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:43,331 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:45,331 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:45,515 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:45,515 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:47,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:47,684 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:47,686 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:40:49,686 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:49,947 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:40:49,947 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 93.5450000763. 2021-12-15 21:40:49,947 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:40:49,950 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:40:50,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:40:50,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:40:50,585 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:40:50,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:40:50,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:40:50,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:40:50,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:40:50,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:40:50,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:40:50,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:40:50,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:51,132 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:51,134 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:40:53,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:53,460 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:40:53,461 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.51399993896. 2021-12-15 21:40:53,463 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:40:53,464 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:40:53,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:40:53,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:40:53,806 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:40:53,806 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:40:53,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:40:53,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:40:53,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:40:53,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:40:53,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:40:53,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:40:53,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:54,092 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:54,092 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:40:56,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:56,737 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:40:56,737 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:40:58,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:40:59,155 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:40:59,157 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.69299983978. ############################################################ # # RAM FootPrint: 4920.14453125 # ############################################################ 2021-12-15 21:40:59,184 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:40:59,184 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 114 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-15 21:42:53,188 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:42:53,188 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:42:58,190 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:42:58,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:42:58,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:42:59,101 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:42:59,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:42:59,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:42:59,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:42:59,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:42:59,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:42:59,204 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:42:59,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:42:59,431 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:42:59,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:42:59,513 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['138252 memcached\n'], []) 2021-12-15 21:42:59,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:42:59,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:42:59,687 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['138252 memcached\n'], []) 2021-12-15 21:42:59,687 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:42:59,691 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:42:59,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:00,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:43:00,562 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:00,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:43:00,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:43:00,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:43:00,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:43:00,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:43:00,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:43:00,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:43:00,999 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:43:01,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:43:01,184 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['94428 memcached\n'], []) 2021-12-15 21:43:01,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:43:01,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:43:01,438 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['94428 memcached\n'], []) 2021-12-15 21:43:01,438 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:43:01,441 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:43:01,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:01,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:43:02,091 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:02,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:43:02,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:43:02,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:43:02,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:43:02,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:43:02,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:43:02,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:43:02,257 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:43:02,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:43:02,346 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['182717 memcached\n'], []) 2021-12-15 21:43:02,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:43:02,415 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:43:02,474 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['182717 memcached\n'], []) 2021-12-15 21:43:02,474 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:43:02,477 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:43:02,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:02,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:43:02,897 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:02,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:43:02,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:43:02,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:43:02,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:43:02,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:43:03,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:43:03,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:43:03,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:43:03,066 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:43:03,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:43:03,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:03,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:03,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:03,306 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:03,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:03,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:03,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:03,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:03,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:03,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:03,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:03,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:03,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:03,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:03,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:03,928 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:03,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:04,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:04,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:04,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:04,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,244 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:04,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:04,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:04,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:04,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:43:04,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:43:04,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:43:04,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:43:04,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:43:04,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:43:04,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:04,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:04,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:43:04,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:43:04,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:43:05,000 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:43:05,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:43:05,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:05,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:43:05,053 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:43:05,053 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:43:05,056 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:43:05,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:05,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:43:05,522 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:05,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:43:05,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:43:05,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:43:05,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:43:05,584 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:43:05,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:43:05,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:43:05,622 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:43:05,637 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:43:05,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:43:05,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:05,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:05,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:05,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:05,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:05,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:05,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:05,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:05,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,068 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:06,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:06,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:06,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:06,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:06,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:06,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:06,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:06,459 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:06,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:06,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:06,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:06,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:06,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:06,734 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:43:06,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:43:06,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:06,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:43:06,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:43:06,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:43:06,918 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:43:06,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:43:06,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:06,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:43:06,954 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:43:06,954 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:43:06,957 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:43:07,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:07,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:43:07,483 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:07,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:43:07,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:43:07,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:43:07,530 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:43:07,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:43:07,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:43:07,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:43:07,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:43:07,599 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:43:07,624 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:43:07,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:07,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:07,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:07,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:43:07,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:07,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:07,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:07,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:43:07,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:07,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:08,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:43:08,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:08,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:43:08,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,262 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:08,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:43:08,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:08,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:43:08,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,510 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:08,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:43:08,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:08,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:43:08,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:43:08,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:08,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:43:08,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:43:08,750 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:43:08,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:43:08,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:43:08,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:43:08,783 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:43:08,785 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:43:08,785 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:43:08,788 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:43:08,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:43:09,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:43:09,299 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:43:09,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:43:09,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:43:09,336 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:43:09,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:43:09,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:43:09,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:43:09,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:43:09,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:09,953 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:43:09,957 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:09,957 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:11,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:12,125 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:12,128 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:12,128 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:14,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:14,295 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:14,299 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:14,301 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:16,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:16,483 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:16,487 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:16,487 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:18,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:18,667 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:18,671 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:18,671 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:20,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:20,851 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:20,854 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:20,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:22,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:23,003 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:23,009 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:23,009 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:25,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:25,187 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:43:25,190 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:25,191 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:27,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:27,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:27,349 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:29,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:29,496 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:29,496 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:31,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:31,644 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:31,644 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:33,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:33,805 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:33,805 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:35,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:35,987 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:35,987 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:37,987 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:38,132 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:38,132 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:40,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:40,299 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:40,299 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:42,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:42,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:42,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:44,451 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:44,627 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:44,627 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:46,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:46,835 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:46,835 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:48,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:49,003 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:49,003 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:51,003 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:51,155 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:51,155 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:53,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:53,309 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:53,309 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:55,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:55,473 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:55,474 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:57,474 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:57,634 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:57,634 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:43:59,635 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:43:59,792 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:43:59,792 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:01,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:01,953 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:01,953 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:03,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:04,114 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:04,115 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:06,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:06,276 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:06,276 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:08,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:08,423 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:08,423 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:10,423 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:10,579 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:10,579 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:12,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:12,740 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:12,740 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:14,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:14,894 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:14,894 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:16,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:17,046 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:17,046 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:19,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:19,209 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:19,209 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:21,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:21,368 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:21,368 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:23,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:23,529 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:23,529 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:25,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:25,684 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:25,684 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:27,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:27,845 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:27,845 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:29,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:30,005 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:30,005 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:32,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:32,164 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:32,164 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:34,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:34,417 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:34,417 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:36,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:36,591 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:36,591 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:38,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:38,773 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:38,773 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:40,775 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:40,967 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:40,967 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:44:42,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:43,224 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:44:43,226 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 94.4409999847. 2021-12-15 21:44:43,226 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:44:43,229 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:44:43,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:44:43,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:44:43,714 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:44:43,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:44:43,727 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:44:43,780 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:44:43,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:44:43,813 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:44:43,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:44:43,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:44:43,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:44,058 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:44,058 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:46,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:46,236 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:46,236 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:48,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:48,407 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:48,407 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:50,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:50,598 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:50,598 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:52,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:52,778 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:52,779 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:54,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:54,963 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:44:54,963 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:44:56,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:57,260 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:44:57,260 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 14.0339999199. 2021-12-15 21:44:57,262 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:44:57,263 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:44:57,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:44:57,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:44:58,075 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:44:58,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:44:58,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:44:58,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:44:58,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:44:58,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:44:58,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:44:58,676 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:44:58,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:44:58,862 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:44:58,862 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.60199999809. ############################################################ # # RAM FootPrint: 4884.453125 # ############################################################ 2021-12-15 21:44:58,891 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:44:58,891 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 80 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-15 21:46:18,895 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:46:18,895 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:46:23,898 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:46:24,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:24,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:46:24,565 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:24,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:46:24,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:46:24,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:46:24,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:46:24,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:46:24,647 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:46:24,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:46:24,687 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:46:24,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:46:24,740 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['139901 memcached\n'], []) 2021-12-15 21:46:24,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:46:24,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:46:24,944 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['139901 memcached\n'], []) 2021-12-15 21:46:24,944 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:46:24,947 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:46:25,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:25,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:46:25,612 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:25,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:46:25,632 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:46:25,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:46:25,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:46:25,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:46:25,790 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:46:25,806 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:46:25,825 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:46:25,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:46:25,895 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['95449 memcached\n'], []) 2021-12-15 21:46:25,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:46:26,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:46:26,252 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['95449 memcached\n'], []) 2021-12-15 21:46:26,253 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:46:26,255 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:46:26,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:26,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:46:26,835 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:26,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:46:26,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:46:26,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:46:26,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:46:26,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:46:26,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:46:26,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:46:26,977 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:46:26,979 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:46:27,062 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['184072 memcached\n'], []) 2021-12-15 21:46:27,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:46:27,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:46:27,216 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['184072 memcached\n'], []) 2021-12-15 21:46:27,217 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:46:27,219 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:46:27,331 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:27,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:46:27,650 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:27,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:46:27,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:46:27,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:46:27,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:46:27,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:46:27,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:46:27,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:46:27,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:46:27,789 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:46:27,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:46:27,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:27,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:27,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:27,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:28,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,068 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:28,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:28,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:28,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:28,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:28,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:28,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:28,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:28,640 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:28,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:28,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:28,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:28,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:28,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:28,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:28,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:29,017 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:29,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:29,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:29,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:29,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:46:29,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:46:29,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:29,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:46:29,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:46:29,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:29,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:46:29,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:46:29,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:29,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:46:29,622 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:46:29,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:46:29,667 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:46:29,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:46:29,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:29,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:46:29,719 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:46:29,719 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:46:29,721 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:46:29,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:30,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:46:30,220 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:30,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:46:30,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:46:30,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:46:30,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:46:30,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:46:30,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:46:30,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:46:30,334 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:46:30,348 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:46:30,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:46:30,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:30,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:30,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:30,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:30,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:30,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:30,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:30,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:30,719 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:30,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:30,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:30,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:30,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:30,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:30,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:30,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:30,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:31,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:31,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:31,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:31,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:31,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:31,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:31,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:31,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,426 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:31,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:31,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:46:31,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:46:31,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:46:31,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:46:31,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:31,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:46:31,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:46:31,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:46:31,713 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:46:31,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:46:31,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:31,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:46:31,752 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:46:31,752 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:46:31,753 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:46:31,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:32,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:46:32,286 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:32,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:46:32,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:46:32,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:46:32,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:46:32,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:46:32,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:46:32,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:46:32,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:46:32,448 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:46:32,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:46:32,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:32,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:46:32,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:32,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:46:32,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:32,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:46:32,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:32,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:46:32,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:32,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:46:32,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:32,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:32,954 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:32,969 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:46:32,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:33,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:33,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:33,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:46:33,063 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:33,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:33,117 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:33,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:46:33,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:33,188 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:46:33,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:33,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:46:33,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:46:33,270 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:46:33,270 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:46:33,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:46:33,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:46:33,316 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:46:33,316 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:46:33,316 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:46:33,319 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:46:33,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:46:33,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:46:33,832 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:46:33,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:46:33,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:46:33,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:46:33,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:46:33,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:46:33,926 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:46:33,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:46:33,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:35,207 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:46:35,211 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:35,213 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:37,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:37,384 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:37,388 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:37,388 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:39,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:39,559 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:39,562 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:39,562 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:41,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:41,730 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:41,733 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:41,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:43,734 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:43,892 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:43,897 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:43,897 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:45,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:46,062 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:46,069 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:46,069 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:48,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:48,247 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:48,253 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:48,253 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:50,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:50,430 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:46:50,434 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:50,434 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:52,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:52,586 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:52,588 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:54,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:54,739 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:54,740 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:56,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:56,885 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:56,885 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:46:58,887 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:46:59,033 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:46:59,033 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:01,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:01,187 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:01,187 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:03,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:03,342 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:03,342 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:05,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:05,483 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:05,483 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:07,484 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:07,632 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:07,632 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:09,632 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:09,864 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:09,864 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:11,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:12,084 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:12,084 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:14,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:14,253 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:14,253 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:16,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:16,403 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:16,403 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:18,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:18,569 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:18,569 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:20,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:20,736 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:20,736 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:22,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:22,895 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:22,895 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:24,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:25,051 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:25,051 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:27,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:27,203 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:27,203 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:29,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:29,358 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:29,358 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:31,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:31,515 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:31,515 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:33,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:33,674 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:33,674 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:35,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:35,838 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:35,838 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:37,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:37,999 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:37,999 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:40,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:40,157 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:40,157 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:42,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:42,309 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:42,309 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:44,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:44,474 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:44,474 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:46,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:46,625 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:46,625 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:48,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:48,792 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:48,792 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:50,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:50,944 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:50,944 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:52,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:53,101 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:53,102 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:55,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:55,263 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:55,263 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:57,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:57,424 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:57,424 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:47:59,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:47:59,594 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:47:59,594 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:48:01,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:01,792 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:48:01,792 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:48:03,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:03,966 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:48:03,966 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:48:05,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:06,130 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:48:06,131 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:48:08,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:08,298 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:48:08,299 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 94.9820001125. 2021-12-15 21:48:08,299 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:48:08,301 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:48:08,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:48:08,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:48:08,785 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:48:08,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:48:08,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:48:08,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:48:08,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:48:08,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:48:08,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:48:08,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:48:08,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:09,095 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:48:09,095 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.796999931335. 2021-12-15 21:48:09,096 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:48:09,098 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:48:09,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:48:09,469 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:48:09,533 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:48:09,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:48:09,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:48:09,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:48:09,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:48:09,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:48:09,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:48:09,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:48:09,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:09,923 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:48:09,924 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-15 21:48:11,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:48:12,414 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:48:12,415 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.31900000572. ############################################################ # # RAM FootPrint: 4844.375 # ############################################################ 2021-12-15 21:48:12,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:48:12,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 77 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-15 21:49:29,456 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:49:29,456 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:49:34,457 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:49:34,913 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:35,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:49:35,693 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:35,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:49:35,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:49:35,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:49:35,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:49:35,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:49:35,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:49:35,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:49:35,984 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:49:35,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:49:36,068 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['141500 memcached\n'], []) 2021-12-15 21:49:36,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:49:36,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:49:36,371 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['141500 memcached\n'], []) 2021-12-15 21:49:36,371 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:49:36,375 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:49:36,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:37,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:49:37,539 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:37,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:49:37,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:49:37,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:49:37,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:49:37,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:49:37,624 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:49:37,635 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:49:37,650 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:49:37,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:49:37,697 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['96518 memcached\n'], []) 2021-12-15 21:49:37,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:49:37,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:49:37,887 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['96518 memcached\n'], []) 2021-12-15 21:49:37,887 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:49:37,890 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:49:38,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:38,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:49:38,369 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:38,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:49:38,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:49:38,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:49:38,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:49:38,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:49:38,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:49:38,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:49:38,536 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:49:38,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:49:38,604 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['185374 memcached\n'], []) 2021-12-15 21:49:38,604 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:49:38,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:49:38,759 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['185374 memcached\n'], []) 2021-12-15 21:49:38,759 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:49:38,762 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:49:38,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:39,122 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:49:39,207 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:39,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:49:39,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:49:39,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:49:39,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:49:39,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:49:39,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:49:39,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:49:39,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:49:39,385 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:49:39,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:49:39,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:39,471 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:39,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:39,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:39,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:39,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:39,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:39,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:39,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:39,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:39,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:39,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:39,788 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:39,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:39,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:39,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:39,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:39,910 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:39,946 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:39,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:39,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,017 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:40,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:40,081 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:40,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:40,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:40,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:40,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:40,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:40,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:40,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:40,545 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,637 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:49:40,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:49:40,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:49:40,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:49:40,813 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,849 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:40,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:49:40,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:49:40,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:40,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:41,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:49:41,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:49:41,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:49:41,053 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:49:41,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:49:41,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:41,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:49:41,104 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:49:41,104 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:49:41,107 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:49:41,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:41,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:49:41,609 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:41,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:49:41,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:49:41,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:49:41,670 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:49:41,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:49:41,704 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:49:41,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:49:41,737 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:49:41,753 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:49:41,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:49:41,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:41,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:41,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:41,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:41,954 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:41,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:42,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:42,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:42,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:42,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,280 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:42,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:42,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:42,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:42,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:42,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:42,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:42,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:42,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:42,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:42,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:42,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:42,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:42,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:43,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:43,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:43,089 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:43,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:43,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:43,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:43,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:49:43,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:49:43,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:43,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:43,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:49:43,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:49:43,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:49:43,352 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:49:43,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:49:43,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:43,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:49:43,400 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:49:43,400 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:49:43,403 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:49:43,510 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:43,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:49:43,921 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:43,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:49:43,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:49:43,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:49:43,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:49:43,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:49:44,003 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:49:44,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:49:44,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:49:44,043 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:49:44,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:49:44,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:44,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:49:44,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:44,362 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:49:44,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:44,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:49:44,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:44,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:49:44,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:44,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:49:44,737 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:44,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:49:44,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:44,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:44,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:44,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:49:44,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:45,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:45,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:45,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:49:45,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:45,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:45,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:45,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:49:45,206 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:45,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:49:45,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:45,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:49:45,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:49:45,289 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:49:45,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:49:45,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:49:45,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:49:45,332 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:49:45,332 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:49:45,332 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:49:45,335 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:49:45,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:49:45,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:49:45,855 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:49:45,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:49:45,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:49:45,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:49:45,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:49:45,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:49:45,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:49:45,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:49:45,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:46,592 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:49:46,596 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:46,596 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:48,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:48,779 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:48,783 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:48,783 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:50,785 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:50,963 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:50,970 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:50,970 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:52,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:53,142 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:53,148 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:53,148 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:55,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:55,319 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:55,323 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:55,323 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:57,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:57,453 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:57,460 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:57,460 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:49:59,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:49:59,640 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:49:59,645 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:49:59,645 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:01,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:01,790 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:01,792 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:03,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:03,944 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:03,944 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:05,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:06,091 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:06,091 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:08,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:08,259 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:08,260 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:10,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:10,415 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:10,417 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:12,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:12,579 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:12,581 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:14,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:14,726 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:14,726 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:16,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:16,865 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:16,865 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:18,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:19,092 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:19,092 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:21,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:21,332 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:21,332 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:23,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:23,497 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:23,497 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:25,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:25,664 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:25,664 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:27,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:27,832 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:27,832 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:29,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:29,989 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:29,989 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:31,989 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:32,148 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:32,148 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:34,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:34,313 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:34,313 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:36,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:36,476 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:36,476 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:38,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:38,648 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:38,648 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:40,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:40,803 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:40,805 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:42,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:42,966 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:42,966 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:44,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:45,124 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:45,124 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:47,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:47,283 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:47,285 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:49,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:49,453 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:49,453 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:51,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:51,611 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:51,611 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:53,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:53,772 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:53,773 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:55,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:55,936 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:55,937 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:50:57,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:50:58,091 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:50:58,091 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:00,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:00,255 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:00,255 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:02,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:02,415 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:02,417 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:04,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:04,573 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:04,575 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:06,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:06,730 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:06,730 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:08,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:08,891 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:08,891 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:10,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:11,157 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:11,157 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:13,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:13,365 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:13,365 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:51:15,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:15,553 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:51:15,555 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.2220001221. 2021-12-15 21:51:15,555 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:51:15,556 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:51:15,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:51:15,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:51:16,121 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:51:16,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:51:16,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:51:16,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:51:16,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:51:16,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:51:16,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:51:16,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:51:16,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:16,457 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:16,457 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:18,459 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:18,663 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:18,663 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:20,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:20,842 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:20,842 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:22,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:23,016 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:23,016 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:25,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:25,191 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:25,193 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:27,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:27,387 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:27,388 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:29,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:29,563 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:51:29,565 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:51:31,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:31,805 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:51:31,805 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 16.2490000725. 2021-12-15 21:51:31,805 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:51:31,808 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:51:32,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:51:33,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:51:33,404 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:51:33,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:51:33,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:51:33,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:51:33,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:51:33,502 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:51:33,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:51:33,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:51:33,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:51:33,752 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:51:33,753 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.9470000267. ############################################################ # # RAM FootPrint: 4829.0859375 # ############################################################ 2021-12-15 21:51:33,779 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:51:33,780 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-15 21:53:20,783 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:53:20,785 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:53:25,786 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:53:26,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:26,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:53:26,484 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:26,484 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:53:26,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:53:26,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:53:26,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:53:26,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:53:26,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:53:26,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:53:26,640 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:53:26,640 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:53:26,724 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['143102 memcached\n'], []) 2021-12-15 21:53:26,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:53:26,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:53:26,855 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['143102 memcached\n'], []) 2021-12-15 21:53:26,855 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:53:26,858 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:53:27,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:27,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:53:27,509 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:27,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:53:27,553 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:53:27,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:53:27,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:53:27,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:53:27,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:53:27,742 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:53:27,773 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:53:27,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:53:27,858 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['97567 memcached\n'], []) 2021-12-15 21:53:27,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:53:27,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:53:28,062 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['97567 memcached\n'], []) 2021-12-15 21:53:28,063 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:53:28,068 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:53:28,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:28,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:53:28,621 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:28,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:53:28,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:53:28,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:53:28,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:53:28,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:53:28,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:53:28,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:53:28,782 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:53:28,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:53:28,848 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['186700 memcached\n'], []) 2021-12-15 21:53:28,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:53:28,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:53:28,999 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['186700 memcached\n'], []) 2021-12-15 21:53:29,000 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:53:29,003 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:53:29,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:29,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:53:29,474 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:29,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:53:29,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:53:29,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:53:29,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:53:29,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:53:29,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:53:29,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:53:29,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:53:29,651 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:53:29,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:53:29,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:29,767 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:29,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:29,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:29,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:29,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:29,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:30,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:30,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:30,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:30,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:30,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:30,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,390 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:30,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:30,446 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:30,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:30,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:30,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:30,727 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,805 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:30,834 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:30,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:30,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:30,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:30,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:30,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:31,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:31,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:31,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:31,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:53:31,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:53:31,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:31,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:53:31,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:53:31,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:31,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:53:31,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:53:31,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:31,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:53:31,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:53:31,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:53:31,655 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:53:31,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:53:31,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:31,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:53:31,710 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:53:31,710 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:53:31,713 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:53:31,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:32,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:53:32,240 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:32,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:53:32,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:53:32,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:53:32,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:53:32,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:53:32,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:53:32,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:53:32,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:53:32,382 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:53:32,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:53:32,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:32,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:32,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:32,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:32,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:32,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:32,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:32,757 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:32,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:32,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:32,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:32,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:32,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:32,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:32,992 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:33,012 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:33,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:33,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:33,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:33,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:33,280 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,364 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:33,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:33,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:33,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:33,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,624 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:33,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:33,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:33,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:33,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:33,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:33,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:53:33,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:53:34,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:34,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:34,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:53:34,105 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:53:34,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:53:34,147 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:53:34,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:53:34,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:34,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:53:34,197 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:53:34,197 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:53:34,200 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:53:34,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:34,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:53:34,707 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:34,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:53:34,727 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:53:34,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:53:34,765 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:53:34,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:53:34,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:53:34,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:53:34,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:53:34,846 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:53:34,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:53:34,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:34,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:35,033 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:53:35,063 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:35,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:53:35,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:35,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:53:35,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:35,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:53:35,471 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:35,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:53:35,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:35,670 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:53:35,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,767 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:35,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:53:35,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:35,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:35,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:53:35,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:35,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:36,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:36,048 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:53:36,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:36,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:53:36,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:36,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:53:36,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:53:36,168 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:53:36,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:53:36,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:53:36,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:53:36,207 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:53:36,207 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:53:36,207 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:53:36,210 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:53:36,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:53:36,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:53:36,707 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:53:36,709 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:53:36,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:53:36,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:53:36,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:53:36,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:53:36,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:53:36,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:53:36,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:37,148 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:53:37,157 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:37,157 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:39,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:39,323 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:39,329 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:39,329 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:41,329 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:41,510 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:41,515 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:41,516 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:43,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:43,693 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:43,697 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:43,697 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:45,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:45,869 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:45,875 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:45,875 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:47,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:48,039 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:48,069 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:48,069 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:50,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:50,230 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:50,234 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:50,234 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:52,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:52,404 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:52,407 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:52,407 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:54,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:54,575 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:54,582 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:54,582 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:56,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:56,739 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:56,743 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:56,743 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:53:58,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:53:58,914 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:53:58,920 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:53:58,920 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:00,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:01,101 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:54:01,105 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:01,107 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:03,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:03,255 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:03,256 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:05,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:05,405 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:05,405 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:07,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:07,553 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:07,553 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:09,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:09,714 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:09,714 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:11,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:11,891 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:11,891 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:13,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:14,036 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:14,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:16,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:16,200 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:16,200 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:18,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:18,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:18,348 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:20,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:20,572 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:20,573 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:22,573 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:22,806 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:22,808 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:24,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:24,971 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:24,973 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:26,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:27,134 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:27,135 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:29,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:29,295 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:29,295 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:31,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:31,453 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:31,454 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:33,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:33,617 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:33,618 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:35,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:35,785 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:35,786 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:37,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:37,953 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:37,954 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:39,954 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:40,127 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:40,127 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:42,127 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:42,279 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:42,279 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:44,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:44,440 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:44,440 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:46,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:46,595 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:46,596 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:48,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:48,749 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:48,750 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:50,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:50,907 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:50,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:52,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:53,069 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:53,069 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:55,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:55,253 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:55,253 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:57,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:57,426 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:57,426 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:54:59,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:54:59,598 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:54:59,598 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:01,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:01,770 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:01,770 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:03,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:03,961 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:03,961 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:05,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:06,128 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:06,130 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:08,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:08,312 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:08,312 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:10,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:10,497 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:10,497 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:12,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:12,678 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:12,678 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:14,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:14,855 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:14,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:55:16,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:17,134 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:55:17,134 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 100.927000046. 2021-12-15 21:55:17,137 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:55:17,138 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:55:17,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:55:17,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:55:17,861 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:55:17,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:55:17,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:55:17,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:55:17,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:55:18,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:55:18,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:55:18,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:55:18,053 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:18,226 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:18,226 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:55:20,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:20,480 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:20,480 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:55:22,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:22,684 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:22,684 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:55:24,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:24,849 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:55:24,851 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:55:26,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:27,276 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:55:27,276 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 10.138999939. 2021-12-15 21:55:27,276 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:55:27,279 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:55:27,423 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:55:27,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:55:27,783 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:55:27,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:55:27,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:55:27,834 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:55:27,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:55:27,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:55:27,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:55:27,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:55:27,969 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:55:28,180 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:55:28,181 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.90499997139. ############################################################ # # RAM FootPrint: 4787.70703125 # ############################################################ 2021-12-15 21:55:28,211 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:55:28,213 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 96 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-15 21:57:04,216 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 21:57:04,217 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 21:57:09,219 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:57:09,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:09,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:57:09,854 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:09,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:57:09,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:57:09,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:57:09,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:57:09,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:57:09,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:57:09,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:57:09,996 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:57:09,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:57:10,075 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['144690 memcached\n'], []) 2021-12-15 21:57:10,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 21:57:10,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 21:57:10,230 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['144690 memcached\n'], []) 2021-12-15 21:57:10,230 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:57:10,233 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:57:10,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:10,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:57:11,540 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:11,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:57:11,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:57:11,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:57:11,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:57:11,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:57:11,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:57:12,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:57:12,400 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:57:12,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:57:12,487 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['98680 memcached\n'], []) 2021-12-15 21:57:12,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 21:57:12,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 21:57:12,740 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['98680 memcached\n'], []) 2021-12-15 21:57:12,740 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:57:12,743 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:57:12,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:13,114 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:57:13,213 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:13,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:57:13,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:57:13,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:57:13,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:57:13,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:57:13,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:57:13,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:57:13,391 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 21:57:13,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:57:13,473 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['188113 memcached\n'], []) 2021-12-15 21:57:13,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 21:57:13,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 21:57:13,628 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['188113 memcached\n'], []) 2021-12-15 21:57:13,628 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:57:13,631 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:57:13,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:13,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:57:14,058 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:14,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:57:14,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:57:14,096 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:57:14,117 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:57:14,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:57:14,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:57:14,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:57:14,186 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:57:14,203 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 21:57:14,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:57:14,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,295 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,336 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:14,362 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:14,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:14,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:14,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,510 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:14,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:14,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:14,665 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:14,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:14,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:14,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:14,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:14,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:14,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:14,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:14,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:14,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:15,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:15,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,105 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:15,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:15,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:15,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:15,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,267 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:57:15,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:57:15,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:57:15,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:57:15,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:57:15,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 21:57:15,459 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:15,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:57:15,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 21:57:15,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:57:15,532 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 21:57:15,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:57:15,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:15,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:57:15,568 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 21:57:15,569 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:57:15,571 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:57:15,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:16,003 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:57:16,084 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:16,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:57:16,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:57:16,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:57:16,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:57:16,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:57:16,165 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:57:16,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:57:16,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:57:16,207 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 21:57:16,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:57:16,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:16,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:16,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:16,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:16,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:16,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:16,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:16,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:16,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:16,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:16,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:16,813 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:16,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:16,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:16,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:16,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:16,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:17,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:17,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,127 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:17,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:17,219 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:17,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:17,336 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:17,443 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:17,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:17,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:17,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:57:17,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 21:57:17,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:17,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:57:17,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 21:57:17,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:57:17,809 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 21:57:17,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:57:17,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:17,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:57:17,848 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 21:57:17,848 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:57:17,851 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:57:17,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:18,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:57:18,355 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:18,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:57:18,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:57:18,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:57:18,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:57:18,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:57:18,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:57:18,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:57:18,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 21:57:18,493 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 21:57:18,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 21:57:18,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:18,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:18,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:18,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 21:57:18,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:18,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:18,806 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:18,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 21:57:18,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:18,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:18,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:18,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 21:57:18,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,028 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:19,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 21:57:19,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:19,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 21:57:19,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:19,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 21:57:19,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:19,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 21:57:19,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:19,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 21:57:19,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:19,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 21:57:19,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 21:57:19,619 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:19,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 21:57:19,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 21:57:19,678 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 21:57:19,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 21:57:19,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 21:57:19,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 21:57:19,730 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 21:57:19,730 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:57:19,730 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:57:19,733 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 21:57:19,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:57:20,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 21:57:20,224 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:57:20,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 21:57:20,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 21:57:20,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 21:57:20,280 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 21:57:20,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 21:57:20,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 21:57:20,329 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 21:57:20,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:20,921 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 21:57:20,924 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:20,924 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:22,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:23,091 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:57:23,095 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:23,095 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:25,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:25,262 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:57:25,267 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:25,269 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:27,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:27,453 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:57:27,457 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:27,457 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:29,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:29,634 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:57:29,641 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:29,641 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:31,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:31,819 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 21:57:31,822 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:31,822 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:33,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:33,980 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:33,980 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:35,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:36,134 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:36,134 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:38,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:38,276 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:38,276 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:40,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:40,430 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:40,430 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:42,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:42,578 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:42,578 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:44,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:44,729 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:44,729 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:46,729 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:46,874 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:46,874 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:48,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:49,022 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:49,022 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:51,022 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:51,247 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:51,247 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:53,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:53,408 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:53,408 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:55,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:55,575 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:55,575 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:57,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:57,732 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:57,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:57:59,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:57:59,891 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:57:59,891 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:01,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:02,059 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:02,059 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:04,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:04,217 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:04,219 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:06,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:06,377 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:06,378 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:08,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:08,535 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:08,535 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:10,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:10,697 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:10,697 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:12,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:12,875 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:12,875 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:14,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:15,036 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:15,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:17,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:17,196 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:17,196 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:19,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:19,351 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:19,352 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:21,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:21,500 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:21,500 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:23,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:23,654 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:23,654 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:25,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:25,811 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:25,811 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:27,811 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:27,977 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:27,977 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:29,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:30,135 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:30,135 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:32,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:32,344 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:32,345 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:34,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:34,526 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:34,526 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:36,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:36,693 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:36,694 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:38,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:38,868 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:38,868 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:40,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:41,049 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:41,049 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:43,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:43,233 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:43,233 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:45,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:45,397 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:45,398 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:47,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:47,565 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:47,565 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 21:58:49,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:49,798 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 21:58:49,799 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.0690000057. 2021-12-15 21:58:49,799 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:58:49,801 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 21:58:49,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:58:50,127 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 21:58:50,186 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:58:50,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 21:58:50,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 21:58:50,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 21:58:50,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 21:58:50,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 21:58:50,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 21:58:50,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 21:58:50,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:50,441 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:50,441 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:58:52,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:52,609 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:52,609 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:58:54,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:54,790 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:54,790 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:58:56,790 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:57,003 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:57,005 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:58:59,005 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:58:59,224 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:58:59,224 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:59:01,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:59:01,427 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 21:59:01,428 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 21:59:03,428 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:59:03,611 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 21:59:03,611 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 13.8120000362. 2021-12-15 21:59:03,614 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 21:59:03,615 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 21:59:03,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 21:59:03,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 21:59:04,013 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 21:59:04,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 21:59:04,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 21:59:04,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 21:59:04,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 21:59:04,109 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 21:59:04,127 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 21:59:04,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 21:59:04,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 21:59:04,352 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 21:59:04,352 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.740999937057. ############################################################ # # RAM FootPrint: 4737.10546875 # ############################################################ 2021-12-15 21:59:04,381 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 21:59:04,381 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 110 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-15 22:00:54,387 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:00:54,387 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:00:59,388 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:00:59,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:00,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:01:00,141 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:00,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:01:00,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:01:00,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:01:00,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:01:00,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:01:00,242 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:01:00,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:01:00,289 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:01:00,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:01:00,361 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['146362 memcached\n'], []) 2021-12-15 22:01:00,362 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:01:00,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:01:00,513 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['146362 memcached\n'], []) 2021-12-15 22:01:00,515 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:01:00,516 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:01:00,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:00,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:01:01,036 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:01,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:01:01,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:01:01,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:01:01,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:01:01,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:01:01,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:01:01,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:01:01,170 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:01:01,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:01:01,233 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['99790 memcached\n'], []) 2021-12-15 22:01:01,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:01:01,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:01:01,319 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['99790 memcached\n'], []) 2021-12-15 22:01:01,321 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:01:01,322 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:01:01,436 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:01,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:01:01,798 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:01,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:01:01,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:01:01,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:01:01,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:01:01,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:01:01,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:01:01,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:01:01,956 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:01:01,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:01:02,019 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['189484 memcached\n'], []) 2021-12-15 22:01:02,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:01:02,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:01:02,124 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['189484 memcached\n'], []) 2021-12-15 22:01:02,124 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:01:02,127 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:01:02,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:02,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:01:02,562 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:02,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:01:02,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:01:02,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:01:02,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:01:02,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:01:02,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:01:02,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:01:02,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:01:02,707 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:01:02,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:01:02,749 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:02,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:02,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:02,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:02,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:02,964 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:03,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:03,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:03,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:03,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:03,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:03,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,492 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:03,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:03,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:03,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:03,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:03,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:03,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:03,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:03,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:03,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:03,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:03,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:03,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:04,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:04,033 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:04,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:04,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:04,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:04,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:04,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:04,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:01:04,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:01:04,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:04,204 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:04,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:01:04,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:01:04,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:04,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:04,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:01:04,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:01:04,364 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:01:04,378 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:01:04,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:01:04,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:04,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:01:04,421 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:01:04,423 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:01:04,424 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:01:04,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:05,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:01:05,470 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:05,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:01:05,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:01:05,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:01:05,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:01:05,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:01:05,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:01:05,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:01:05,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:01:05,614 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:01:05,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:01:05,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:05,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:05,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:05,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:05,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:05,829 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:05,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:05,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:05,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:05,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:05,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:05,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:06,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,048 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:06,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:06,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:06,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:06,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:06,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:06,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:06,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:06,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,436 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:06,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:06,474 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,522 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:06,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:06,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:06,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:06,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:06,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:06,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:01:06,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:01:06,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,770 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:06,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:01:06,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:01:06,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:01:06,815 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:01:06,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:01:06,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:06,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:01:06,848 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:01:06,848 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:01:06,849 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:01:06,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:07,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:01:07,388 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:07,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:01:07,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:01:07,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:01:07,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:01:07,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:01:07,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:01:07,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:01:07,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:01:07,552 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:01:07,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:01:07,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:07,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:07,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:07,729 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:01:07,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:07,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:07,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:07,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:01:07,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:07,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:07,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:08,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:01:08,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:08,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:01:08,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:08,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:01:08,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:08,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:01:08,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:08,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:01:08,505 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:08,604 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:01:08,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:08,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:01:08,755 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:08,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:01:08,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:08,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:01:08,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:08,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:01:08,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:01:08,974 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:01:08,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:01:08,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:01:09,017 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:01:09,036 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:01:09,036 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:01:09,036 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:01:09,039 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:01:09,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:01:09,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:01:09,519 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:01:09,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:01:09,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:01:09,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:01:09,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:01:09,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:01:09,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:01:09,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:01:09,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:11,210 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 22:01:11,214 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:11,214 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:13,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:13,374 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:13,378 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:13,378 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:15,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:15,555 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:15,559 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:15,559 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:17,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:17,746 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:17,752 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:17,752 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:19,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:19,903 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:19,907 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:19,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:21,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:22,023 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:22,028 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:22,029 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:24,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:24,197 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:24,201 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:24,201 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:26,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:26,388 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:01:26,394 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:26,394 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:28,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:28,546 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:28,546 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:30,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:30,703 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:30,703 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:32,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:32,855 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:32,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:34,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:35,006 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:35,006 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:37,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:37,161 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:37,161 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:39,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:39,311 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:39,311 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:41,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:41,464 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:41,464 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:43,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:43,625 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:43,625 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:45,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:45,858 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:45,858 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:47,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:48,055 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:48,056 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:50,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:50,217 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:50,217 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:52,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:52,372 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:52,372 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:54,372 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:54,532 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:54,532 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:56,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:56,693 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:56,693 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:01:58,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:01:58,858 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:01:58,858 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:00,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:01,013 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:01,013 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:03,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:03,170 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:03,170 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:05,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:05,335 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:05,335 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:07,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:07,486 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:07,486 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:09,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:09,641 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:09,641 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:11,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:11,806 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:11,806 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:13,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:13,967 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:13,967 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:15,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:16,128 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:16,128 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:18,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:18,295 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:18,295 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:20,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:20,453 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:20,453 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:22,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:22,619 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:22,619 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:24,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:25,051 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:25,052 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:27,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:27,227 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:27,227 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:29,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:29,391 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:29,392 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:31,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:31,575 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:31,575 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:33,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:33,749 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:33,749 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:35,749 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:35,907 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:35,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:37,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:38,098 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:38,098 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:40,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:40,282 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:40,282 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:42,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:42,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:02:42,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:02:44,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:45,654 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:02:45,655 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.6199998856. 2021-12-15 22:02:45,655 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:02:45,658 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:02:46,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:02:50,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:02:50,604 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:02:50,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:02:50,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:02:50,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:02:50,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:02:50,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:02:50,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:02:50,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:02:50,946 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:51,098 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:02:51,098 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.4430000782. 2021-12-15 22:02:51,099 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:02:51,101 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:02:51,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:02:51,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:02:51,500 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:02:51,502 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:02:51,525 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:02:51,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:02:51,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:02:51,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:02:51,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:02:51,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:02:51,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:02:51,835 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:02:51,835 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.735000133514. ############################################################ # # RAM FootPrint: 4569.53125 # ############################################################ 2021-12-15 22:02:51,871 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:02:51,871 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-15 22:04:43,875 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:04:43,875 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:04:48,877 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:04:49,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:49,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:04:49,581 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:49,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:04:49,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:04:49,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:04:49,640 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:04:49,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:04:49,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:04:49,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:04:49,719 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:04:49,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:04:49,789 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['148001 memcached\n'], []) 2021-12-15 22:04:49,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:04:49,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:04:49,888 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['148001 memcached\n'], []) 2021-12-15 22:04:49,888 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:04:49,890 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:04:50,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:50,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:04:50,621 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:50,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:04:50,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:04:50,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:04:50,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:04:50,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:04:50,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:04:50,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:04:50,857 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:04:50,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:04:50,918 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['100960 memcached\n'], []) 2021-12-15 22:04:50,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:04:51,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:04:51,128 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['100960 memcached\n'], []) 2021-12-15 22:04:51,128 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:04:51,131 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:04:51,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:51,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:04:51,591 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:51,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:04:51,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:04:51,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:04:51,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:04:51,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:04:51,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:04:51,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:04:51,760 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:04:51,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:04:51,846 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['190872 memcached\n'], []) 2021-12-15 22:04:51,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:04:51,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:04:51,989 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['190872 memcached\n'], []) 2021-12-15 22:04:51,989 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:04:51,992 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:04:52,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:52,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:04:52,444 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:52,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:04:52,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:04:52,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:04:52,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:04:52,525 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:04:52,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:04:52,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:04:52,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:04:52,592 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:04:52,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:04:52,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:52,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:52,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:52,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:52,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:52,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:52,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:52,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:52,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:53,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:53,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:53,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:53,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,339 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:53,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:53,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:53,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:53,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:53,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:53,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:53,767 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:53,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:53,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:53,877 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:53,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:53,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:53,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:53,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:54,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:54,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:54,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:54,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:54,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:04:54,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:04:54,117 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:54,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:54,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:04:54,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:04:54,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:54,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:54,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:04:54,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:04:54,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:04:54,283 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:04:54,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:04:54,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:54,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:04:54,323 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:04:54,325 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:04:54,326 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:04:54,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:54,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:04:54,838 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:54,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:04:54,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:04:54,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:04:54,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:04:54,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:04:54,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:04:54,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:04:54,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:04:54,960 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:04:54,989 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:04:55,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:55,142 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:55,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:55,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:55,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:55,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:55,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,517 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:55,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:55,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,653 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:55,701 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:55,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,765 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:55,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:55,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:55,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:55,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:55,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:55,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:55,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:56,017 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,043 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:56,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:56,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:56,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:56,176 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:56,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:56,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,244 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:56,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:56,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:56,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:56,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:04:56,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:04:56,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:56,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:04:56,446 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:04:56,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:04:56,480 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:04:56,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:04:56,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:56,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:04:56,525 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:04:56,525 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:04:56,526 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:04:56,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:56,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:04:57,039 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:57,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:04:57,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:04:57,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:04:57,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:04:57,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:04:57,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:04:57,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:04:57,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:04:57,164 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:04:57,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:04:57,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:57,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:04:57,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:57,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:04:57,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:57,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:04:57,502 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:57,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:04:57,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,614 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:57,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:04:57,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,721 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:57,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:04:57,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:57,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:04:57,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:57,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:57,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:04:57,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:57,989 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:58,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:58,025 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:04:58,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:58,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:58,081 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:58,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:04:58,109 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:58,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:04:58,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:58,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:04:58,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:04:58,173 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:04:58,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:04:58,183 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:04:58,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:04:58,204 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:04:58,204 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:04:58,204 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:04:58,207 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:04:58,323 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:04:59,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:04:59,249 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:04:59,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:04:59,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:04:59,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:04:59,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:04:59,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:04:59,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:04:59,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:04:59,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:00,668 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 22:05:00,673 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:00,673 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:02,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:02,858 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:02,864 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:02,865 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:04,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:05,029 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:05,035 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:05,035 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:07,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:07,194 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:07,200 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:07,200 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:09,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:09,375 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:09,380 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:09,380 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:11,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:11,559 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:11,563 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:11,563 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:13,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:13,744 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:05:13,749 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:13,749 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:15,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:15,894 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:15,894 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:17,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:18,043 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:18,043 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:20,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:20,191 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:20,191 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:22,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:22,355 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:22,355 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:24,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:24,512 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:24,513 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:26,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:26,667 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:26,668 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:28,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:28,812 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:28,812 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:30,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:30,963 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:30,963 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:32,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:33,164 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:33,164 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:35,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:35,391 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:35,392 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:37,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:37,551 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:37,552 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:39,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:39,710 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:39,710 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:41,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:41,871 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:41,871 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:43,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:44,036 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:44,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:46,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:46,188 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:46,190 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:48,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:48,352 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:48,354 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:50,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:50,513 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:50,513 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:52,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:52,680 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:52,680 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:54,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:54,846 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:54,848 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:56,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:57,003 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:57,003 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:05:59,005 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:05:59,163 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:05:59,164 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:01,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:01,323 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:01,325 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:03,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:03,487 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:03,487 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:05,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:05,647 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:05,648 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:07,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:07,806 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:07,806 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:09,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:09,973 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:09,973 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:11,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:12,134 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:12,134 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:14,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:14,296 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:14,298 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:16,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:16,667 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:16,667 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:18,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:18,845 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:18,845 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:20,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:21,045 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:21,045 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:23,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:23,216 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:23,216 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:25,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:25,404 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:25,405 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:27,405 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:27,565 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:27,565 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:06:29,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:29,763 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:06:29,763 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 91.5580000877. 2021-12-15 22:06:29,765 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:06:29,766 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:06:29,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:06:30,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:06:30,233 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:06:30,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:06:30,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:06:30,270 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:06:30,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:06:30,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:06:30,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:06:30,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:06:30,415 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:30,598 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:30,598 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:06:32,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:32,776 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:06:32,776 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:06:34,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:34,987 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:06:34,987 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.22299981117. 2021-12-15 22:06:34,989 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:06:34,990 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:06:35,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:06:35,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:06:35,946 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:06:35,946 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:06:35,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:06:35,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:06:36,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:06:36,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:06:36,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:06:36,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:06:36,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:06:36,315 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:06:36,315 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.32699990273. ############################################################ # # RAM FootPrint: 4542.61328125 # ############################################################ 2021-12-15 22:06:36,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:06:36,348 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 100 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-15 22:08:16,352 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:08:16,352 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:08:21,355 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:08:21,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:21,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:08:21,959 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:21,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:08:21,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:08:22,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:08:22,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:08:22,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:08:22,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:08:22,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:08:22,088 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:08:22,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:08:22,177 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['149668 memcached\n'], []) 2021-12-15 22:08:22,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:08:22,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:08:22,329 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['149668 memcached\n'], []) 2021-12-15 22:08:22,329 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:08:22,332 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:08:22,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:22,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:08:22,986 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:22,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:08:23,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:08:23,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:08:23,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:08:23,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:08:23,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:08:23,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:08:23,292 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:08:23,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:08:23,382 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['102045 memcached\n'], []) 2021-12-15 22:08:23,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:08:23,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:08:23,605 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['102045 memcached\n'], []) 2021-12-15 22:08:23,605 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:08:23,608 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:08:23,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:23,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:08:24,069 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:24,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:08:24,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:08:24,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:08:24,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:08:24,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:08:24,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:08:24,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:08:24,211 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:08:24,211 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:08:24,266 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['192238 memcached\n'], []) 2021-12-15 22:08:24,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:08:24,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:08:24,420 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['192238 memcached\n'], []) 2021-12-15 22:08:24,420 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:08:24,421 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:08:24,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:24,788 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:08:24,877 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:24,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:08:24,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:08:24,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:08:24,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:08:24,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:08:24,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:08:24,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:08:25,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:08:25,029 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:08:25,053 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:08:25,081 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:25,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:25,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:25,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:25,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,517 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:25,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:25,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,599 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:25,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:25,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:25,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:25,816 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:25,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:25,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:25,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:25,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:26,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:26,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:26,209 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:26,234 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,270 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,306 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:26,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:26,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:26,471 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:26,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:26,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:26,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:08:26,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:08:26,719 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,790 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:08:26,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:08:26,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:26,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:08:26,928 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:08:26,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:08:26,966 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:08:26,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:08:26,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:26,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:08:27,012 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:08:27,012 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:08:27,015 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:08:27,117 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:27,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:08:27,506 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:27,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:08:27,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:08:27,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:08:27,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:08:27,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:08:27,584 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:08:27,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:08:27,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:08:27,622 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:08:27,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:08:27,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:27,711 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:27,752 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:27,785 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:27,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:27,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:27,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:27,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:27,974 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:28,086 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:28,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:28,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:28,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:28,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:28,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:28,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:28,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:28,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:28,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,670 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:28,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:28,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:28,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:28,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:28,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:28,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:28,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:28,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:29,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:29,025 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:29,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:29,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:08:29,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:08:29,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:29,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:29,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:08:29,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:08:29,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:08:29,168 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:08:29,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:08:29,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:29,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:08:29,203 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:08:29,203 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:08:29,206 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:08:29,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:29,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:08:29,700 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:29,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:08:29,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:08:29,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:08:29,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:08:29,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:08:29,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:08:29,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:08:29,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:08:29,838 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:08:29,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:08:29,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:29,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:29,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:29,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:08:29,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:29,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:30,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:08:30,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:30,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:08:30,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,163 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:30,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:08:30,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:30,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:08:30,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,323 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:30,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:08:30,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:30,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:08:30,436 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,459 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:30,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:08:30,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:30,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:08:30,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:30,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:08:30,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:08:30,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:30,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:08:30,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:08:30,703 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:08:30,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:08:30,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:08:30,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:08:30,733 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:08:30,733 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:08:30,734 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:08:30,736 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:08:30,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:08:31,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:08:31,216 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:08:31,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:08:31,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:08:31,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:08:31,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:08:31,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:08:31,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:08:31,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:08:31,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:32,701 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 22:08:32,707 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:32,707 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:34,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:34,844 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:08:34,851 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:34,851 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:36,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:37,036 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:08:37,040 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:37,040 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:39,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:39,223 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:08:39,226 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:39,226 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:41,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:41,398 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:08:41,403 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:41,403 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:43,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:43,573 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:08:43,578 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:43,578 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:45,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:45,729 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:45,730 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:47,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:47,881 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:47,881 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:49,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:50,033 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:50,033 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:52,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:52,180 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:52,180 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:54,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:54,326 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:54,326 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:56,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:56,473 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:56,473 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:08:58,474 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:08:58,621 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:08:58,621 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:00,622 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:00,796 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:00,796 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:02,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:02,941 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:02,941 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:04,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:05,171 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:05,171 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:07,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:07,328 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:07,328 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:09,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:09,484 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:09,484 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:11,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:11,648 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:11,648 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:13,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:13,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:13,802 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:15,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:15,961 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:15,961 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:17,961 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:18,127 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:18,127 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:20,127 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:20,275 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:20,275 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:22,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:22,444 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:22,444 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:24,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:24,607 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:24,607 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:26,607 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:26,767 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:26,767 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:28,767 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:28,918 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:28,918 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:30,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:31,084 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:31,084 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:33,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:33,247 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:33,249 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:35,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:35,410 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:35,410 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:37,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:37,569 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:37,569 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:39,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:39,729 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:39,730 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:41,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:41,898 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:41,898 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:43,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:44,065 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:44,065 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:46,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:46,236 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:46,237 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:48,237 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:48,401 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:48,401 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:50,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:50,585 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:50,585 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:52,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:52,780 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:52,780 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:54,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:54,971 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:54,971 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:56,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:57,144 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:57,145 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:09:59,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:09:59,325 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:09:59,325 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:10:01,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:10:01,568 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:10:01,569 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.8340001106. 2021-12-15 22:10:01,569 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:10:01,572 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:10:01,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:10:01,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:10:02,042 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:10:02,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:10:02,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:10:02,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:10:02,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:10:02,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:10:02,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:10:02,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:10:02,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:10:02,369 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:10:02,369 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:10:04,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:10:04,542 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:10:04,543 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:10:06,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:10:07,303 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:10:07,303 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.73399996758. 2021-12-15 22:10:07,305 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:10:07,306 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:10:07,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:10:07,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:10:07,658 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:10:07,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:10:07,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:10:07,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:10:07,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:10:07,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:10:07,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:10:07,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:10:07,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:10:08,007 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:10:08,009 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.705000162125. ############################################################ # # RAM FootPrint: 4518.79296875 # ############################################################ 2021-12-15 22:10:08,042 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:10:08,042 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-15 22:11:54,046 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:11:54,046 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:11:59,049 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:11:59,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:11:59,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:11:59,674 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:11:59,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:11:59,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:11:59,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:11:59,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:11:59,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:11:59,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:11:59,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:11:59,786 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:11:59,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:11:59,871 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['151273 memcached\n'], []) 2021-12-15 22:11:59,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:11:59,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:12:00,017 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['151273 memcached\n'], []) 2021-12-15 22:12:00,017 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:12:00,020 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:12:00,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:00,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:12:00,621 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:00,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:12:00,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:12:00,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:12:00,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:12:00,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:12:00,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:12:00,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:12:00,763 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:12:00,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:12:00,859 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['103124 memcached\n'], []) 2021-12-15 22:12:00,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:12:00,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:12:00,996 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['103124 memcached\n'], []) 2021-12-15 22:12:00,996 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:12:01,000 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:12:01,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:01,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:12:01,638 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:01,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:12:01,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:12:01,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:12:01,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:12:01,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:12:01,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:12:01,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:12:01,812 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:12:01,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:12:01,901 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['193603 memcached\n'], []) 2021-12-15 22:12:01,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:12:01,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:12:02,036 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['193603 memcached\n'], []) 2021-12-15 22:12:02,036 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:12:02,039 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:12:02,150 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:02,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:12:02,417 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:02,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:12:02,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:12:02,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:12:02,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:12:02,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:12:02,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:12:02,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:12:02,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:12:02,566 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:12:02,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:12:02,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:02,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:02,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:02,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:02,727 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:02,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:02,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:02,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:02,829 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:02,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:02,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:02,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:02,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:02,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:02,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:02,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:02,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:03,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:03,066 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,086 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:03,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:03,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:03,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:03,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:03,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:03,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:03,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:03,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,505 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:03,530 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:03,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:03,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:03,640 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:03,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:03,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,747 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,769 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:12:03,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:12:03,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:12:03,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:12:03,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:03,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:03,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:12:03,992 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:12:04,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:12:04,025 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:12:04,025 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:12:04,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:04,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:12:04,072 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:12:04,073 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:12:04,075 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:12:04,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:04,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:12:04,575 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:04,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:12:04,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:12:04,614 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:12:04,635 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:12:04,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:12:04,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:12:04,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:12:04,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:12:04,713 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:12:04,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:12:04,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:04,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:04,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:04,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:04,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:04,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:05,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:05,063 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:05,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:05,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,234 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:05,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:05,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:05,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:05,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:05,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:05,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:05,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:05,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:05,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:05,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:05,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:05,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:05,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:05,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:05,910 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:05,928 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:05,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:05,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:05,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:06,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:06,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:06,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:12:06,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:12:06,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:12:06,117 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:12:06,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:12:06,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:06,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:12:06,161 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:12:06,161 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:12:06,164 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:12:06,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:06,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:12:06,686 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:06,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:12:06,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:12:06,724 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:12:06,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:12:06,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:12:06,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:12:06,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:12:06,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:12:06,831 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:12:06,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:12:06,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:06,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:06,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:07,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:12:07,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:07,165 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:12:07,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:07,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:12:07,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:07,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:12:07,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,479 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:07,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:12:07,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:07,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:12:07,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,678 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:07,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:12:07,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:07,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:12:07,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:07,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:12:07,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:07,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:07,979 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:07,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:12:08,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:08,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:08,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:08,081 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:12:08,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:08,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:12:08,121 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:08,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:12:08,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:12:08,154 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:12:08,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:12:08,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:12:08,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:12:08,187 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:12:08,187 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:12:08,187 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:12:08,190 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:12:08,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:12:10,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:12:10,621 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:12:10,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:12:10,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:12:10,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:12:10,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:12:10,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:12:10,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:12:10,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:12:10,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:10,904 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:10,911 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:10,911 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:12,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:13,055 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:13,059 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:13,059 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:15,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:15,164 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:15,167 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:15,167 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:17,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:17,345 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:17,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:17,348 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:19,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:19,532 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:19,536 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:19,536 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:21,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:21,719 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:12:21,723 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:21,723 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:23,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:23,865 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:23,865 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:25,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:26,035 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:26,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:28,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:28,190 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:28,190 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:30,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:30,339 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:30,339 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:32,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:32,489 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:32,490 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:34,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:34,645 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:34,645 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:36,647 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:36,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:36,801 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:38,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:39,003 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:39,003 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:41,005 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:41,219 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:41,220 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:43,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:43,385 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:43,385 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:45,387 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:45,540 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:45,540 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:47,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:47,698 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:47,698 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:49,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:49,858 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:49,858 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:51,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:52,013 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:52,013 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:54,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:54,184 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:54,184 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:56,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:56,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:56,348 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:12:58,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:12:58,509 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:12:58,509 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:00,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:00,663 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:00,663 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:02,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:02,826 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:02,826 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:04,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:04,986 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:04,986 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:06,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:07,148 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:07,148 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:09,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:09,312 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:09,313 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:11,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:11,467 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:11,469 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:13,469 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:13,628 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:13,628 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:15,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:15,786 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:15,786 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:17,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:17,941 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:17,943 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:19,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:20,098 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:20,098 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:22,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:22,259 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:22,259 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:24,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:24,773 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:24,773 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:26,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:26,951 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:26,951 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:28,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:29,131 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:29,131 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:31,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:31,311 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:31,311 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:33,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:33,655 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:33,655 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:35,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:35,924 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:35,926 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:37,926 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:38,098 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:38,099 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:13:40,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:40,374 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:13:40,374 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 92.1870000362. 2021-12-15 22:13:40,375 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:13:40,377 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:13:40,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:13:40,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:13:40,806 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:13:40,806 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:13:40,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:13:40,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:13:40,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:13:40,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:13:40,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:13:40,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:13:41,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:41,178 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:41,180 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:13:43,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:43,361 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:13:43,361 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:13:45,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:45,688 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:13:45,688 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.31399989128. 2021-12-15 22:13:45,690 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:13:45,691 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:13:45,982 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:13:46,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:13:46,424 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:13:46,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:13:46,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:13:46,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:13:46,484 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:13:46,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:13:46,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:13:46,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:13:46,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:13:46,878 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:13:46,878 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.18799996376. ############################################################ # # RAM FootPrint: 4499.78125 # ############################################################ 2021-12-15 22:13:46,907 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:13:46,907 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 79 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-15 22:15:05,911 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:15:05,911 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:15:10,914 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:15:11,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:11,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:15:12,160 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:12,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:15:12,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:15:12,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:15:12,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:15:12,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:15:12,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:15:12,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:15:12,384 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:15:12,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:15:12,460 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['152825 memcached\n'], []) 2021-12-15 22:15:12,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:15:12,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:15:12,729 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['152825 memcached\n'], []) 2021-12-15 22:15:12,729 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:15:12,732 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:15:12,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:13,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:15:13,276 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:13,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:15:13,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:15:13,368 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:15:13,390 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:15:13,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:15:13,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:15:13,446 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:15:13,461 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:15:13,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:15:13,519 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['104265 memcached\n'], []) 2021-12-15 22:15:13,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:15:13,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:15:13,723 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['104265 memcached\n'], []) 2021-12-15 22:15:13,723 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:15:13,726 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:15:13,839 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:14,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:15:14,151 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:14,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:15:14,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:15:14,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:15:14,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:15:14,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:15:14,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:15:14,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:15:14,283 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:15:14,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:15:14,332 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['195000 memcached\n'], []) 2021-12-15 22:15:14,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:15:14,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:15:14,477 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['195000 memcached\n'], []) 2021-12-15 22:15:14,479 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:15:14,482 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:15:14,586 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:14,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:15:14,921 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:14,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:15:14,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:15:14,961 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:15:14,979 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:15:14,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:15:15,012 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:15:15,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:15:15,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:15:15,056 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:15:15,096 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:15:15,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:15,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:15,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:15,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:15,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:15,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:15,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:15,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:15,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:15,624 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:15,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:15,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:15,775 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:15,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:15,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:15,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:15,961 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,002 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:16,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:16,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:16,170 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:16,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:16,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:16,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,372 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:16,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:16,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:16,516 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:16,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,573 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,607 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:16,632 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:16,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,688 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:16,765 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:16,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:15:16,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:15:16,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:16,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:16,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:15:16,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:15:17,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:17,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:17,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:15:17,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:15:17,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:15:17,105 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:15:17,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:15:17,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:17,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:15:17,150 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:15:17,150 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:15:17,151 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:15:17,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:17,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:15:17,650 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:17,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:15:17,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:15:17,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:15:17,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:15:17,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:15:17,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:15:17,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:15:17,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:15:17,796 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:15:17,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:15:17,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:17,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:17,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:17,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:17,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:18,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:18,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,229 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:18,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:18,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,336 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:18,362 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:18,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:18,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:18,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:18,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:18,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:18,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:18,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:18,698 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:18,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:18,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:18,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:18,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:18,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:18,908 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:18,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:18,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:18,959 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:18,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:18,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:19,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:19,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:15:19,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:15:19,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:15:19,061 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:15:19,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:15:19,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:19,082 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:15:19,092 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:15:19,094 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:15:19,096 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:15:19,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:19,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:15:19,625 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:19,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:15:19,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:15:19,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:15:19,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:15:19,697 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:15:19,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:15:19,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:15:19,742 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:15:19,759 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:15:19,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:15:19,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:19,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:19,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:19,931 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:15:19,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:20,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:15:20,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,209 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:20,237 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:15:20,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:20,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:15:20,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:20,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:15:20,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:20,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:15:20,622 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,693 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:20,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:15:20,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,760 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,788 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:20,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:15:20,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,880 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:20,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:15:20,913 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:20,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:20,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:20,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:15:20,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:21,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:21,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:21,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:15:21,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:21,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:15:21,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:21,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:15:21,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:15:21,141 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:15:21,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:15:21,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:15:21,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:15:21,174 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:15:21,174 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:15:21,174 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:15:21,177 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:15:21,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:15:21,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:15:21,661 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:15:21,661 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:15:21,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:15:21,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:15:21,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:15:21,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:15:21,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:15:21,772 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:15:21,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:23,150 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-15 22:15:23,154 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:23,154 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 Some events could not be published because the queue was (likely temporarily) over capacity: {"SaslMechanismsListedEvent":{"sampleEvent":"SaslMechanismsListedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0.073869269S, createdAt=30276394521263219, description=Listed SASL mechanisms: [SCRAM_SHA512, SCRAM_SHA256, SCRAM_SHA1, PLAIN], context={bucket=GleamBookUsers0, coreId=0xd97b4671000000ef, local=/172.23.120.105:36928, remote=172.23.110.65/172.23.110.65:11210}, cause=null}","totalDropCount":1},"SelectBucketFailedEvent":{"sampleEvent":"SelectBucketFailedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0S, createdAt=30276394519287901, description=Select bucket failed with status code 0x1, context={bucket=GleamBookUsers0, coreId=0xd97b4671000000f5, local=/172.23.120.105:36882, remote=172.23.110.65/172.23.110.65:11210}, cause=null}","totalDropCount":12},"RequestRetryScheduledEvent":{"sampleEvent":"RequestRetryScheduledEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0.016S, createdAt=30276394521961228, description=Request UpsertRequest retry scheduled per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context={completed=false, coreId=0xd97b4671000000cf, idempotent=false, lastDispatchedTo=172.23.110.64, requestId=9412271299, requestType=UpsertRequest, retried=6, retryReasons=[ENDPOINT_NOT_AVAILABLE], service={bucket=GleamBookUsers0, collection=VolumeCollection42, documentId=bLoRzCC1-114762710, opaque=0x310a4246, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null}","totalDropCount":974},"RequestNotRetriedEvent":{"sampleEvent":"RequestNotRetriedEvent{severity=DEBUG, category=com.couchbase.request, duration=PT0S, createdAt=30276394513617568, description=Request CarrierGlobalConfigRequest not retried per RetryStrategy (Reason: NODE_NOT_AVAILABLE), context={cancelled=true, completed=true, coreId=0xd97b4671000000e4, idempotent=true, reason=NO_MORE_RETRIES (NODE_NOT_AVAILABLE), requestId=9412271996, requestType=CarrierGlobalConfigRequest, retried=0, service={opaque=0x310a45d5, target=172.23.110.64, type=kv}, timeoutMs=2500}, cause=null}","totalDropCount":1},"SaslAuthenticationCompletedEvent":{"sampleEvent":"SaslAuthenticationCompletedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0.077712074S, createdAt=30276394519269490, description=SASL Authentication completed, context={bucket=GleamBookUsers0, coreId=0xd97b4671000000f5, local=/172.23.120.105:36882, remote=172.23.110.65/172.23.110.65:11210}, cause=null}","totalDropCount":12},"EndpointConnectionFailedEvent":{"sampleEvent":"EndpointConnectionFailedEvent{severity=WARN, category=com.couchbase.endpoint, duration=PT0.081585687S, createdAt=30276394521825511, description=Connect attempt 1 failed because of : finishConnect(..) failed: Connection refused: /172.23.110.64:11210 - Check server ports and cluster encryption setting., context={bucket=GleamBookUsers0, channelId=D97B4671000000DD/00000000FC0C7F12, circuitBreaker=DISABLED, coreId=0xd97b4671000000dd, local=172.23.120.105:33686, remote=172.23.110.64:11210, type=KV}, cause=com.couchbase.client.core.endpoint.BaseEndpoint$2: finishConnect(..) failed: Connection refused: /172.23.110.64:11210 - Check server ports and cluster encryption setting.}","totalDropCount":13},"EndpointStateChangedEvent":{"sampleEvent":"EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=30276394521779086, description=Endpoint changed state from CONNECTING to DISCONNECTED, context={bucket=GleamBookUsers0, channelId=D97B4671000000DD/00000000FC0C7F12, circuitBreaker=DISABLED, coreId=0xd97b4671000000dd, local=172.23.120.105:33686, remote=172.23.110.64:11210, type=KV}, cause=null}","totalDropCount":13},"FeaturesNegotiatedEvent":{"sampleEvent":"FeaturesNegotiatedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0.07360677S, createdAt=30276394520981723, description=Negotiated [MUTATION_SEQNO, XATTR, XERROR, SELECT_BUCKET, SNAPPY, UNORDERED_EXECUTION, TRACING, ALT_REQUEST, SYNC_REPLICATION, COLLECTIONS, PRESERVE_TTL, VATTR, CREATE_AS_DELETED], context={bucket=GleamBookUsers0, coreId=0xd97b4671000000ef, local=/172.23.120.105:36928, remote=172.23.110.65/172.23.110.65:11210}, cause=null}","totalDropCount":1},"ServiceStateChangedEvent":{"sampleEvent":"ServiceStateChangedEvent{severity=DEBUG, category=com.couchbase.service, duration=PT0S, createdAt=30276394521790896, description=Service changed state from CONNECTING to DISCONNECTED, context={bucket=GleamBookUsers0, coreId=0xd97b4671000000dd, remote=172.23.110.64:11210, type=KV}, cause=null}","totalDropCount":13},"SaslMechanismsSelectedEvent":{"sampleEvent":"SaslMechanismsSelectedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0S, createdAt=30276394520317869, description=SASL Mechanism SCRAM_SHA512 selected. Allowed: [SCRAM_SHA1, SCRAM_SHA256, SCRAM_SHA512], context={bucket=GleamBookUsers0, coreId=0xd97b4671000000df, local=/172.23.120.105:35884, remote=172.23.110.64/172.23.110.64:11210}, cause=null}","totalDropCount":21},"ErrorMapLoadedEvent":{"sampleEvent":"ErrorMapLoadedEvent{severity=DEBUG, category=com.couchbase.io, duration=PT0.073738189S, createdAt=30276394521216044, description=KV Error Map successfully loaded. Map Version: 1, Revision: 5, context={bucket=GleamBookUsers0, coreId=0xd97b4671000000ef, local=/172.23.120.105:36928, remote=172.23.110.65/172.23.110.65:11210}, cause=null}","totalDropCount":1}} 2021-12-15 22:15:25,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:25,273 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:15:25,279 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:25,279 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:27,280 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:27,450 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:15:27,454 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:27,454 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:29,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:29,640 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:15:29,644 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:29,644 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:31,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:31,812 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:15:31,816 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:31,816 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:33,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:34,009 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:15:34,013 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:34,013 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:36,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:36,161 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:36,161 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:38,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:38,312 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:38,312 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:40,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:40,463 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:40,463 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:42,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:42,618 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:42,618 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:44,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:44,775 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:44,775 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:46,775 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:46,926 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:46,927 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:48,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:49,073 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:49,073 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:51,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:51,253 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:51,253 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:53,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:53,457 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:53,459 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:55,459 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:55,694 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:55,694 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:57,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:15:57,855 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:15:57,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:15:59,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:00,020 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:00,020 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:02,020 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:02,180 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:02,181 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:04,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:04,342 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:04,342 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:06,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:06,497 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:06,497 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:08,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:08,657 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:08,657 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:10,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:10,821 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:10,821 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:12,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:12,982 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:12,982 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:14,982 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:15,138 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:15,138 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:17,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:17,302 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:17,302 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:19,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:19,467 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:19,467 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:21,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:21,618 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:21,619 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:23,619 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:23,786 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:23,786 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:25,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:25,951 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:25,953 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:27,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:28,101 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:28,101 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:30,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:30,259 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:30,259 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:32,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:32,418 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:32,418 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:34,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:34,855 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:34,855 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:36,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:37,188 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:37,188 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:39,188 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:39,375 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:39,377 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:41,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:41,549 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:41,549 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:43,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:43,763 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:43,763 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:45,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:45,937 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:45,937 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:47,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:48,115 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:48,115 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:16:50,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:50,545 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:16:50,545 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 89.370000124. 2021-12-15 22:16:50,546 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:16:50,548 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:16:50,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:16:50,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:16:50,970 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:16:50,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:16:50,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:16:51,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:16:51,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:16:51,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:16:51,068 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:16:51,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:16:51,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:51,355 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:51,357 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:16:53,357 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:53,535 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:16:53,535 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:16:55,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:55,698 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:16:55,700 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.15400004387. 2021-12-15 22:16:55,700 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:16:55,701 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:16:56,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:16:56,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:16:56,335 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:16:56,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:16:56,357 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:16:56,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:16:56,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:16:56,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:16:56,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:16:56,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:16:56,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:16:56,709 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:16:56,709 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.00800013542. ############################################################ # # RAM FootPrint: 4465.37109375 # ############################################################ 2021-12-15 22:16:56,743 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:16:56,743 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 116 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-15 22:18:52,746 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:18:52,746 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:18:57,749 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:18:58,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:18:58,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:18:58,414 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:18:58,415 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:18:58,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:18:58,456 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:18:58,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:18:58,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:18:58,515 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:18:58,529 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:18:58,546 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:18:58,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:18:58,628 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['154484 memcached\n'], []) 2021-12-15 22:18:58,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:18:58,704 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:18:58,778 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['154484 memcached\n'], []) 2021-12-15 22:18:58,778 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:18:58,780 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:18:58,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:18:59,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:18:59,403 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:18:59,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:18:59,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:18:59,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:18:59,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:18:59,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:18:59,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:18:59,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:18:59,575 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:18:59,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:18:59,664 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['105354 memcached\n'], []) 2021-12-15 22:18:59,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:18:59,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:18:59,960 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['105354 memcached\n'], []) 2021-12-15 22:18:59,961 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:18:59,963 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:19:00,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:19:00,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:19:00,440 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:19:00,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:19:00,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:19:00,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:19:00,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:19:00,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:19:00,543 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:19:00,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:19:00,575 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:19:00,575 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:19:00,644 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['196347 memcached\n'], []) 2021-12-15 22:19:00,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:19:00,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:19:00,782 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['196347 memcached\n'], []) 2021-12-15 22:19:00,782 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:19:00,786 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:19:00,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:19:01,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:19:01,184 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:19:01,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:19:01,206 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:19:01,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:19:01,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:19:01,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:19:01,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:19:01,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:19:01,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:19:01,339 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:19:01,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:19:01,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:01,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:01,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:01,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:01,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:01,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:01,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:01,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:01,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,834 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:01,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:01,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:01,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:01,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:01,961 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:01,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:01,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:02,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:02,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:02,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:02,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:02,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:02,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:02,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:02,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:02,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:02,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:02,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:19:02,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:19:02,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:19:02,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:19:02,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:02,604 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:19:02,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:19:02,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:19:02,647 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:19:02,647 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:19:02,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:02,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:19:02,687 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:19:02,687 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:19:02,690 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:19:02,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:19:03,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:19:03,190 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:19:03,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:19:03,204 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:19:03,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:19:03,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:19:03,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:19:03,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:19:03,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:19:03,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:19:03,305 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:19:03,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:19:03,352 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:03,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:03,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:03,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:03,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:03,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:03,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:03,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:03,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:03,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:03,782 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:03,806 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:03,826 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:03,867 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:03,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:03,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:03,941 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:03,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:04,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:04,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,130 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:04,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:04,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:04,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:04,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:04,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:04,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:04,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:04,578 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,614 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,637 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:04,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:04,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:04,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:04,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,811 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:04,846 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:04,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,910 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:19:04,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:19:04,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:04,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:04,964 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:19:04,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:19:04,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:19:04,997 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:19:04,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:19:05,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:05,020 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:19:05,032 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:19:05,032 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:19:05,033 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:19:05,148 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:19:09,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:19:09,744 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:19:09,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:19:09,763 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:19:09,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:19:09,799 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:19:09,818 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:19:09,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:19:09,846 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:19:09,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:19:09,878 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:19:09,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:19:09,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:09,979 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:10,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:19:10,066 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:10,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:19:10,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,331 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:10,357 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:19:10,380 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,423 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:10,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:19:10,484 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:10,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:19:10,589 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,637 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:10,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:19:10,732 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,788 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:10,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:19:10,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:10,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:10,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:11,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:19:11,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:11,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:11,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:11,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:19:11,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:11,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:11,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:11,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:19:11,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:11,309 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:11,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:11,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:19:11,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:11,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:19:11,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:11,479 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:19:11,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:19:11,513 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:19:11,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:19:11,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:19:11,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:19:11,563 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:19:11,563 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:19:11,563 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:19:11,569 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:19:11,696 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:19:11,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:19:12,055 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:19:12,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:19:12,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:19:12,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:19:12,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:19:12,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:19:12,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:19:12,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:19:12,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:12,322 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:19:12,326 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:12,326 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:14,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:14,483 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:19:14,487 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:14,487 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:16,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:16,667 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:19:16,671 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:16,671 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:18,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:18,854 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:19:18,858 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:18,858 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:20,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:21,026 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:19:21,030 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:21,030 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:23,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:23,171 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:23,173 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:25,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:25,321 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:25,321 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:27,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:27,463 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:27,463 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:29,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:29,628 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:29,628 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:31,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:31,798 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:31,798 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:33,798 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:33,947 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:33,947 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:35,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:36,098 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:36,099 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:38,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:38,246 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:38,246 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:40,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:40,400 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:40,401 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:42,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:42,589 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:42,589 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:44,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:44,759 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:44,759 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:46,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:46,918 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:46,920 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:48,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:49,078 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:49,078 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:51,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:51,232 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:51,233 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:53,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:53,395 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:53,397 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:55,397 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:55,548 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:55,548 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:57,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:57,713 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:57,713 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:19:59,713 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:19:59,871 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:19:59,871 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:01,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:02,036 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:02,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:04,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:04,196 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:04,196 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:06,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:06,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:06,349 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:08,349 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:08,509 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:08,509 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:10,509 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:10,671 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:10,673 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:12,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:12,829 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:12,829 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:14,829 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:14,993 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:14,993 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:16,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:17,157 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:17,157 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:19,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:19,319 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:19,319 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:21,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:21,483 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:21,483 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:23,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:23,641 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:23,642 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:25,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:25,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:25,802 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:27,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:27,971 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:27,971 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:29,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:30,168 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:30,168 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:32,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:32,345 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:32,346 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:34,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:34,513 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:34,513 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:36,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:36,684 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:36,684 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:20:38,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:38,851 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:20:38,852 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.2890000343. 2021-12-15 22:20:38,854 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:20:38,855 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:20:38,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:20:39,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:20:39,252 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:20:39,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:20:39,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:20:39,275 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:20:39,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:20:39,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:20:39,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:20:39,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:20:39,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:39,542 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:39,542 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:41,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:41,694 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:41,694 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:43,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:44,194 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:44,194 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:46,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:46,401 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:46,401 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:48,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:48,881 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:48,881 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:50,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:51,061 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:51,061 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:53,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:53,243 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:53,243 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:55,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:55,404 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:20:55,404 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:20:57,404 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:20:59,805 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:20:59,806 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 20.9529998302. 2021-12-15 22:20:59,808 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:20:59,809 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:20:59,963 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:21:00,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:21:00,328 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:21:00,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:21:00,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:21:00,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:21:00,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:21:00,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:21:00,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:21:00,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:21:00,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:21:00,665 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:21:00,665 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.857999801636. ############################################################ # # RAM FootPrint: 4446.90625 # ############################################################ 2021-12-15 22:21:00,700 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:21:00,700 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 70 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-15 22:22:10,704 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:22:10,706 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:22:15,707 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:22:15,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:16,288 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:22:16,359 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:16,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:22:16,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:22:16,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:22:16,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:22:16,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:22:16,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:22:16,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:22:16,545 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:22:16,545 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:22:16,621 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['156056 memcached\n'], []) 2021-12-15 22:22:16,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:22:16,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:22:16,750 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['156056 memcached\n'], []) 2021-12-15 22:22:16,750 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:22:16,753 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:22:16,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:18,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:22:18,280 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:18,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:22:18,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:22:18,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:22:18,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:22:18,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:22:18,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:22:18,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:22:18,507 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:22:18,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:22:18,601 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['106126 memcached\n'], []) 2021-12-15 22:22:18,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:22:18,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:22:18,793 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['106126 memcached\n'], []) 2021-12-15 22:22:18,795 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:22:18,796 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:22:18,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:19,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:22:19,276 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:19,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:22:19,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:22:19,321 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:22:19,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:22:19,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:22:19,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:22:19,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:22:19,415 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:22:19,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:22:19,473 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['197771 memcached\n'], []) 2021-12-15 22:22:19,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:22:19,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:22:19,628 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['197771 memcached\n'], []) 2021-12-15 22:22:19,628 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:22:19,631 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:22:19,737 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:19,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:22:20,078 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:20,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:22:20,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:22:20,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:22:20,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:22:20,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:22:20,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:22:20,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:22:20,211 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:22:20,229 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:22:20,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:22:20,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:20,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:20,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:20,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:20,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:20,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:20,553 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:20,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:20,621 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:20,676 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:20,720 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:20,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:20,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:20,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:20,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:20,917 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:20,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:20,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,032 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:21,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:21,089 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:21,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:21,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,249 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:21,296 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:21,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:21,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:21,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,492 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:21,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:21,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:21,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:21,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:21,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:21,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:21,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:21,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:21,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:21,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:22,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:22,043 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:22:22,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:22:22,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:22,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:22,167 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:22:22,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:22:22,221 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:22,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:22,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:22:22,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:22:22,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:22:22,339 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:22:22,339 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:22:22,357 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:22,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:22:22,391 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:22:22,391 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:22:22,394 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:22:22,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:22,829 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:22:22,904 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:22,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:22:22,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:22:22,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:22:22,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:22:22,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:22:23,000 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:22:23,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:22:23,038 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:22:23,055 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:22:23,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:22:23,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:23,227 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:23,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:23,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:23,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,476 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:23,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:23,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:23,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:23,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:23,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:23,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,706 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:23,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:23,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:23,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:23,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:23,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:23,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:23,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:24,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,051 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:24,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:24,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:24,280 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:24,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:24,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:24,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:24,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:24,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,562 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:22:24,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:22:24,618 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,632 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:24,653 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:22:24,665 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:22:24,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:22:24,690 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:22:24,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:22:24,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:24,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:22:24,726 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:22:24,726 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:22:24,729 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:22:24,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:25,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:22:25,253 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:25,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:22:25,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:22:25,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:22:25,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:22:25,334 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:22:25,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:22:25,364 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:22:25,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:22:25,394 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:22:25,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:22:25,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:25,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:25,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:25,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:22:25,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:25,648 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:25,691 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:25,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:22:25,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:25,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:25,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:25,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:22:25,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:25,908 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:25,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:25,957 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:22:25,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:25,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,020 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:26,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:22:26,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:26,105 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:22:26,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:26,174 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:22:26,190 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:26,239 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:22:26,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:26,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:22:26,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:26,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:22:26,384 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:26,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:22:26,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,467 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:22:26,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:26,496 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:22:26,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:22:26,517 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:22:26,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:22:26,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:22:26,539 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:22:26,549 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:22:26,551 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:22:26,551 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:22:26,553 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:22:26,677 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:22:26,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:22:27,076 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:22:27,076 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:22:27,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:22:27,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:22:27,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:22:27,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:22:27,170 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:22:27,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:22:27,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:27,354 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:22:27,358 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:27,358 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:29,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:29,533 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:22:29,539 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:29,539 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:31,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:31,684 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:22:31,688 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:31,688 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:33,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:33,859 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:22:33,864 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:33,864 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:35,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:36,040 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:22:36,045 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:36,045 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:38,046 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:38,191 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:38,191 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:40,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:40,345 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:40,345 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:42,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:42,496 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:42,497 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:44,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:44,650 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:44,650 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:46,650 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:46,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:46,801 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:48,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:48,948 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:48,948 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:50,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:51,102 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:51,102 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:53,104 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:53,259 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:53,259 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:55,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:55,471 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:55,471 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:57,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:57,698 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:57,698 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:22:59,700 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:22:59,862 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:22:59,864 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:01,864 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:02,025 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:02,025 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:04,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:04,181 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:04,181 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:06,181 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:06,338 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:06,338 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:08,338 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:08,505 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:08,505 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:10,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:10,674 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:10,674 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:12,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:12,828 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:12,828 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:14,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:14,986 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:14,986 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:16,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:17,142 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:17,142 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:19,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:19,292 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:19,292 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:21,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:21,450 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:21,450 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:23,451 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:23,615 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:23,615 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:25,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:25,786 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:25,786 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:27,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:27,951 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:27,951 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:29,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:30,118 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:30,118 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:32,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:32,282 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:32,282 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:34,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:34,446 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:34,446 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:36,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:36,602 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:36,602 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:38,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:38,789 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:38,789 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:40,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:40,960 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:40,960 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:42,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:43,137 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:43,137 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:45,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:45,319 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:45,319 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:47,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:47,492 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:47,492 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:49,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:49,665 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:49,665 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:23:51,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:51,825 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:23:51,826 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 85.2759997845. 2021-12-15 22:23:51,828 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:23:51,828 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:23:51,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:23:52,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:23:52,286 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:23:52,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:23:52,312 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:23:52,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:23:52,410 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:23:52,426 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:23:52,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:23:52,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:23:52,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:52,657 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:52,657 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:23:54,657 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:54,842 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:54,842 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:23:56,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:57,016 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:57,016 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:23:59,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:23:59,204 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:23:59,206 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:24:01,206 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:24:01,392 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:24:01,392 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:24:03,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:24:03,621 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:24:03,621 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 11.7939999104. 2021-12-15 22:24:03,621 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:24:03,624 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:24:03,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:24:03,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:24:04,055 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:24:04,055 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:24:04,141 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:24:04,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:24:04,170 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:24:04,187 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:24:04,198 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:24:04,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:24:04,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:24:04,380 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:24:04,380 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.758000135422. ############################################################ # # RAM FootPrint: 4435.0 # ############################################################ 2021-12-15 22:24:04,408 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:24:04,408 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-15 22:25:22,413 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:25:22,413 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:25:27,414 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:25:27,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:28,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:25:28,381 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:28,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:25:28,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:25:28,487 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:25:28,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:25:28,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:25:28,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:25:28,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:25:28,581 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:25:28,581 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:25:28,653 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['157675 memcached\n'], []) 2021-12-15 22:25:28,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:25:28,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:25:28,865 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['157675 memcached\n'], []) 2021-12-15 22:25:28,865 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:25:28,868 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:25:29,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:29,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:25:29,753 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:29,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:25:29,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:25:29,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:25:29,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:25:29,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:25:30,056 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:25:30,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:25:30,098 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:25:30,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:25:30,184 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['107663 memcached\n'], []) 2021-12-15 22:25:30,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:25:30,276 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:25:30,413 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['107663 memcached\n'], []) 2021-12-15 22:25:30,413 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:25:30,417 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:25:30,548 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:30,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:25:30,882 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:30,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:25:30,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:25:30,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:25:30,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:25:30,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:25:30,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:25:31,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:25:31,026 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:25:31,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:25:31,085 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['199172 memcached\n'], []) 2021-12-15 22:25:31,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:25:31,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:25:31,246 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['199172 memcached\n'], []) 2021-12-15 22:25:31,246 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:25:31,250 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:25:31,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:31,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:25:31,694 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:31,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:25:31,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:25:31,734 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:25:31,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:25:31,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:25:31,795 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:25:31,811 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:25:31,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:25:31,846 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:25:31,871 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:25:31,895 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:31,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:31,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:32,032 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:32,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:32,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:32,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,290 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:32,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:32,414 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,502 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:32,538 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:32,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,609 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:32,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:32,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,755 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:32,832 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:32,862 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:32,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:32,940 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:32,969 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:32,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,030 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:33,092 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:33,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,145 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:33,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:33,214 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,237 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:33,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:33,298 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:33,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:33,394 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,417 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,443 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:33,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:33,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:25:33,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:25:33,552 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:25:33,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:25:33,624 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:33,665 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:25:33,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:25:33,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:25:33,707 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:25:33,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:25:33,719 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:33,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:25:33,744 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:25:33,744 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:25:33,747 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:25:33,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:36,048 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:25:36,135 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:36,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:25:36,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:25:36,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:25:36,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:25:36,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:25:36,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:25:36,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:25:36,265 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:25:36,282 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:25:36,306 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:25:36,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:36,377 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:36,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:36,461 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:36,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:36,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:36,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:36,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:36,665 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:36,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:36,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:36,780 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:36,808 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:36,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:36,881 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:36,910 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:36,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:36,979 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:37,032 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:37,058 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:37,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:37,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,260 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:37,289 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:37,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,385 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:37,415 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:37,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:37,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:37,601 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,684 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:37,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:37,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:37,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:37,885 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:37,930 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:37,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:37,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:38,026 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:38,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:38,115 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:25:38,138 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:25:38,154 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:38,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:38,197 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:25:38,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:25:38,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:25:38,234 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:25:38,234 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:25:38,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:38,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:25:38,275 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:25:38,276 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:25:38,278 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:25:38,390 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:38,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:25:38,819 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:38,821 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:25:38,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:25:38,859 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:25:38,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:25:38,904 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:25:38,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:25:38,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:25:38,951 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:25:38,967 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:25:38,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:25:39,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,078 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,119 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:39,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:25:39,188 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,288 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:39,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:25:39,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,400 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,430 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:39,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:25:39,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:39,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:25:39,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:39,717 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:25:39,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,815 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:39,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:25:39,869 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:39,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:39,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:39,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:25:40,015 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,061 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:40,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:40,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:25:40,151 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:40,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:40,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:25:40,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,319 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:40,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:40,378 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:25:40,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,454 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:40,494 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:40,520 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:25:40,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:25:40,627 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:40,658 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:25:40,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:25:40,707 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:25:40,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:25:40,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:25:40,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:25:40,759 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:25:40,759 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:25:40,760 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:25:40,763 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:25:40,885 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:25:41,142 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:25:41,256 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:25:41,256 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:25:41,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:25:41,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:25:41,323 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:25:41,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:25:41,364 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:25:41,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:25:41,398 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:41,598 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:41,605 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:41,605 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:43,605 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:43,789 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:43,795 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:43,796 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:45,796 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:45,971 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:45,976 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:45,976 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:47,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:48,128 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:48,132 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:48,132 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:50,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:50,312 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:50,315 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:50,315 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:52,316 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:52,499 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:25:52,503 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:52,503 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:54,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:54,651 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:54,651 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:56,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:56,801 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:56,801 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:25:58,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:25:58,947 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:25:58,947 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:00,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:01,096 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:01,096 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:03,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:03,246 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:03,246 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:05,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:05,394 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:05,395 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:07,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:07,540 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:07,540 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:09,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:09,759 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:09,759 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:11,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:11,993 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:11,993 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:13,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:14,167 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:14,167 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:16,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:16,325 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:16,326 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:18,326 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:18,482 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:18,482 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:20,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:20,655 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:20,655 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:22,655 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:22,821 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:22,822 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:24,822 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:24,983 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:24,983 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:26,984 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:27,145 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:27,145 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:29,147 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:29,322 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:29,322 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:31,322 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:31,471 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:31,471 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:33,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:33,635 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:33,637 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:35,637 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:35,793 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:35,795 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:37,795 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:37,947 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:37,948 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:39,948 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:40,094 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:40,094 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:42,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:42,270 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:42,270 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:44,270 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:44,424 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:44,424 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:46,424 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:46,582 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:46,582 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:48,582 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:48,743 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:48,744 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:50,744 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:50,888 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:50,888 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:52,888 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:53,040 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:53,040 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:55,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:55,196 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:55,196 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:57,196 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:57,351 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:57,351 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:26:59,351 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:26:59,507 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:26:59,507 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:27:01,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:01,703 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:01,703 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:27:03,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:03,898 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:03,898 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:27:05,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:06,059 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:06,059 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:27:08,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:08,223 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:08,223 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:27:10,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:10,437 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:27:10,437 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 89.6779999733. 2021-12-15 22:27:10,438 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:27:10,440 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:27:10,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:27:10,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:27:10,829 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:27:10,831 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:27:10,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:27:10,854 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:27:10,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:27:10,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:27:10,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:27:10,905 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:27:10,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:11,073 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:11,073 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:13,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:13,234 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:13,234 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:15,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:15,848 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:15,848 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:17,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:18,095 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:18,095 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:20,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:20,266 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:20,266 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:22,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:22,437 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:27:22,437 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:27:24,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:24,615 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:27:24,615 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 14.1770000458. 2021-12-15 22:27:24,617 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:27:24,618 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:27:24,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:27:25,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:27:25,970 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:27:25,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:27:26,105 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:27:26,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:27:26,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:27:26,161 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:27:26,216 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:27:26,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:27:26,247 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:27:26,483 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:27:26,483 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.86599993706. ############################################################ # # RAM FootPrint: 4394.22265625 # ############################################################ 2021-12-15 22:27:26,510 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:27:26,510 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 71 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-15 22:28:37,515 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:28:37,515 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:28:42,517 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:28:42,634 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:42,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:28:43,131 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:43,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:28:43,164 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:28:43,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:28:43,301 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:28:43,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:28:43,339 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:28:43,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:28:43,371 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:28:43,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:28:43,440 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['159278 memcached\n'], []) 2021-12-15 22:28:43,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:28:43,630 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:28:43,752 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['159278 memcached\n'], []) 2021-12-15 22:28:43,752 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:28:43,755 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:28:43,996 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:44,877 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:28:45,019 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:45,020 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:28:45,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:28:45,111 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:28:45,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:28:45,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:28:45,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:28:45,206 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:28:45,223 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:28:45,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:28:45,285 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['108694 memcached\n'], []) 2021-12-15 22:28:45,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:28:45,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:28:45,395 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['108694 memcached\n'], []) 2021-12-15 22:28:45,395 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:28:45,398 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:28:45,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:45,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:28:45,903 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:45,903 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:28:45,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:28:45,950 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:28:45,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:28:46,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:28:46,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:28:46,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:28:46,071 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:28:46,071 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:28:46,131 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['200535 memcached\n'], []) 2021-12-15 22:28:46,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:28:46,211 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:28:46,282 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['200535 memcached\n'], []) 2021-12-15 22:28:46,282 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:28:46,285 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:28:46,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:46,644 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:28:46,740 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:46,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:28:46,765 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:28:46,786 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:28:46,812 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:28:46,836 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:28:46,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:28:46,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:28:46,892 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:28:46,911 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:28:46,944 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:28:46,970 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,016 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:47,098 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:47,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:47,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:47,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:47,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:47,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,420 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:47,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:47,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,542 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:47,592 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:47,614 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,641 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,664 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:47,687 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:47,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,756 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:47,779 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:47,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:47,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:47,897 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:47,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:47,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:47,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:47,986 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,010 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:48,063 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:48,086 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,109 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:48,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:48,176 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,236 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:48,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:48,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,342 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:48,361 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:48,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,411 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:28:48,451 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:28:48,469 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,528 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:28:48,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:28:48,563 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:48,596 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 22:28:48,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 22:28:48,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:28:48,638 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:28:48,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:28:48,653 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:48,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:28:48,684 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:28:48,684 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:28:48,686 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:28:48,803 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:49,124 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:28:49,207 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:49,207 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:28:49,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:28:49,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:28:49,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:28:49,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:28:49,299 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:28:49,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:28:49,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:28:49,349 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:28:49,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:28:49,403 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:49,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:49,497 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:49,533 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:49,565 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:49,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:49,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:49,681 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:49,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:49,743 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:49,773 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:49,801 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:49,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:49,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:49,884 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:49,910 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:49,933 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:49,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:49,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:50,002 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:50,023 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:50,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:50,107 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,134 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:50,177 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:50,194 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:50,286 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:50,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:50,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:50,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,413 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:50,448 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:50,463 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,483 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:50,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:50,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:50,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:50,611 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,635 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:50,668 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:50,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,703 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:50,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:28:50,740 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:28:50,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:28:50,766 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:28:50,766 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:28:50,778 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:50,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:28:50,803 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:28:50,803 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:28:50,805 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:28:50,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:51,253 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:28:51,335 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:51,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:28:51,354 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:28:51,369 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:28:51,390 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:28:51,408 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:28:51,423 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:28:51,437 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:28:51,450 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:28:51,466 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:28:51,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:28:51,536 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:51,594 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:51,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:51,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:28:51,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:51,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:51,785 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:51,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:28:51,846 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:51,890 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:51,918 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:51,946 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:28:51,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,039 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:52,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:28:52,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,140 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:52,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:28:52,230 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:52,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:28:52,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,469 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:52,499 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:28:52,523 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,566 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:52,638 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:28:52,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,714 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:52,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:28:52,809 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:52,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:52,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:28:52,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:52,987 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:53,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:53,059 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:28:53,088 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:53,131 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:53,157 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:53,180 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:28:53,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:53,223 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:28:53,240 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:53,255 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:28:53,269 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:28:53,280 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:28:53,282 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:28:53,293 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:28:53,306 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:28:53,319 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:28:53,319 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:28:53,319 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:28:53,322 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:28:53,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:28:53,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:28:53,819 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:28:53,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:28:53,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:28:53,857 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:28:53,878 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:28:53,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:28:53,927 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:28:53,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:28:53,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:28:54,164 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:28:54,171 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:28:54,171 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:28:56,173 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:28:56,351 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:28:56,359 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:28:56,359 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:28:58,359 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:28:58,467 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:28:58,473 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:28:58,473 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:00,473 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:00,637 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:29:00,641 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:00,641 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:02,642 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:02,812 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:29:02,816 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:02,818 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:04,818 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:04,970 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:29:04,974 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:04,974 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:06,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:07,125 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:07,125 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:09,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:09,276 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:09,276 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:11,278 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:11,428 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:11,428 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:13,428 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:13,588 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:13,588 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:15,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:15,740 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:15,742 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:17,742 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:17,891 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:17,891 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:19,891 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:20,036 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:20,036 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:22,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:22,200 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:22,200 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:24,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:24,421 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:24,421 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:26,421 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:26,584 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:26,584 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:28,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:28,730 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:28,730 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:30,730 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:30,898 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:30,898 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:32,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:33,069 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:33,069 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:35,069 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:35,233 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:35,233 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:37,233 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:37,394 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:37,394 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:39,395 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:39,558 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:39,558 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:41,558 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:41,717 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:41,717 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:43,719 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:43,875 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:43,875 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:45,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:46,035 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:46,035 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:48,035 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:48,198 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:48,198 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:50,200 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:50,355 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:50,355 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:52,355 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:52,512 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:52,512 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:54,512 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:54,671 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:54,671 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:56,673 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:56,828 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:56,828 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:29:58,828 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:29:58,992 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:29:58,992 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:00,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:01,154 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:01,155 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:03,155 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:03,316 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:03,318 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:05,318 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:05,480 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:05,480 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:07,480 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:07,635 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:07,635 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:09,637 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:09,846 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:09,848 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:11,848 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:12,019 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:12,019 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:14,019 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:14,210 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:14,211 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:16,211 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:16,374 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:16,374 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:18,374 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:18,586 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:18,588 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:20,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:20,752 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:20,753 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:30:22,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:22,888 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:30:22,888 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 89.5690000057. 2021-12-15 22:30:22,890 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:30:22,891 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:30:22,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:30:23,607 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:30:23,736 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:30:23,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:30:23,776 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:30:23,802 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:30:23,838 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:30:23,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:30:23,885 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:30:23,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:30:23,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:24,084 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:24,084 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:30:26,084 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:26,265 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:26,265 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:30:28,266 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:28,447 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:28,447 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:30:30,447 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:30,625 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:30:30,625 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:30:32,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:32,838 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:30:32,838 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 9.94899988174. 2021-12-15 22:30:32,839 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:30:32,842 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:30:33,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:30:33,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:30:34,273 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:30:34,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:30:34,344 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:30:34,375 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:30:34,388 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:30:34,482 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:30:34,517 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:30:34,546 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:30:34,559 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:30:34,700 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:30:34,700 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.86100006104. ############################################################ # # RAM FootPrint: 4396.265625 # ############################################################ 2021-12-15 22:30:34,730 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-15 22:30:34,730 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 80 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-15 22:31:54,733 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-15 22:31:54,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-15 22:31:59,736 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:31:59,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:00,273 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:32:00,345 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:00,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:32:00,418 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:32:00,443 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:32:00,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:32:00,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:32:00,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:32:00,517 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:32:00,532 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:32:00,532 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:32:00,588 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['160925 memcached\n'], []) 2021-12-15 22:32:00,588 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-15 22:32:00,667 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-15 22:32:00,767 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['160925 memcached\n'], []) 2021-12-15 22:32:00,767 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:32:00,769 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:32:00,983 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:01,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:32:01,427 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:01,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:32:01,441 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:32:01,519 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:32:01,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:32:01,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:32:01,561 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:32:01,572 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:32:01,585 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:32:01,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:32:01,632 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['109980 memcached\n'], []) 2021-12-15 22:32:01,632 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-15 22:32:01,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-15 22:32:01,855 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['109980 memcached\n'], []) 2021-12-15 22:32:01,857 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:32:01,858 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:32:02,033 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:02,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:32:02,405 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:02,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:32:02,434 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:32:02,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:32:02,486 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:32:02,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:32:02,535 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:32:02,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:32:02,579 | infra | DEBUG | Thread-2 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 22:32:02,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:32:02,674 | infra | DEBUG | Thread-2 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['201941 memcached\n'], []) 2021-12-15 22:32:02,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 22:32:02,757 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 22:32:02,828 | infra | DEBUG | Thread-2 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['201941 memcached\n'], []) 2021-12-15 22:32:02,828 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:32:02,831 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:32:02,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:03,158 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:32:03,237 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:03,239 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:32:03,257 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:32:03,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:32:03,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:32:03,328 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:32:03,346 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:32:03,364 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:32:03,382 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:32:03,403 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 22:32:03,433 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:32:03,457 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:03,506 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:03,556 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:03,595 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:03,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:03,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:03,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:03,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:03,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:03,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:03,934 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:03,971 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:04,005 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,040 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,073 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:04,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:04,125 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,201 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:04,232 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:04,259 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,303 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,335 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:04,365 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:04,392 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,426 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,466 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:04,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:04,513 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,549 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:04,608 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:04,630 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:04,739 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:04,765 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,789 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:04,851 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:04,872 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:04,915 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:04,947 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:04,973 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:04,999 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:05,068 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:05,094 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:05,118 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,144 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:05,168 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:05,193 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:05,213 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,246 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:05,285 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:32:05,313 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:32:05,341 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:05,401 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:32:05,436 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 22:32:05,464 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:05,510 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 22:32:05,530 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 22:32:05,551 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:32:05,568 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 22:32:05,568 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:32:05,585 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:05,602 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:32:05,619 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 22:32:05,619 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:32:05,622 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:32:05,736 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:06,052 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:32:06,137 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:06,137 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:32:06,160 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:32:06,178 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:32:06,203 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:32:06,226 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:32:06,244 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:32:06,262 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:32:06,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:32:06,299 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 22:32:06,323 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:32:06,345 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:06,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:06,453 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:06,489 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:06,522 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:06,579 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:06,625 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:06,663 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:06,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:06,746 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:06,792 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:06,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:06,855 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:06,901 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:06,938 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:06,967 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:06,993 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,036 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,072 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:07,102 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:07,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,210 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:07,239 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:07,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,305 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,334 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:07,358 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:07,381 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,427 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,470 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:07,500 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:07,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,573 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,617 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:07,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:07,671 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,710 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,737 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:07,762 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:07,783 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,819 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:07,868 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:07,887 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:07,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:07,956 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:07,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:08,013 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:08,062 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:08,099 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:08,128 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:08,153 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:08,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:08,217 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:32:08,243 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 22:32:08,263 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:32:08,279 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 22:32:08,279 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:32:08,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:08,308 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:32:08,321 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 22:32:08,321 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:32:08,323 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:32:08,431 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:08,742 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:32:08,825 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:08,825 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:32:08,842 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:32:08,858 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:32:08,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:32:08,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:32:08,907 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:32:08,920 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:32:08,936 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 22:32:08,950 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-15 22:32:08,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 22:32:08,997 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,045 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,095 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:09,132 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 22:32:09,163 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,191 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,220 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:09,250 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 22:32:09,272 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,292 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,315 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:09,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 22:32:09,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,371 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,391 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:09,407 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 22:32:09,423 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,460 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:09,474 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 22:32:09,490 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,507 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:09,540 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 22:32:09,555 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,576 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:09,612 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 22:32:09,628 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,651 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,676 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:09,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 22:32:09,704 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,723 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,753 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:09,775 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 22:32:09,793 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,823 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,845 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:09,861 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 22:32:09,875 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,894 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,914 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:09,928 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 22:32:09,943 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:09,960 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:09,980 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:09,994 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 22:32:10,009 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:10,028 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 22:32:10,042 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:10,053 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 22:32:10,065 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 22:32:10,075 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-15 22:32:10,075 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 22:32:10,085 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 22:32:10,096 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 22:32:10,107 | test | DEBUG | Thread-2 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-15 22:32:10,107 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:32:10,107 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:32:10,109 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 22:32:10,224 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:32:10,525 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 22:32:10,598 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:32:10,598 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 22:32:10,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 22:32:10,631 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 22:32:10,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 22:32:10,674 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 22:32:10,690 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 22:32:10,707 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 22:32:10,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:10,914 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:10,921 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:10,921 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:12,921 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:13,082 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:13,089 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:13,089 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:15,091 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:15,267 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:15,273 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:15,273 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:17,275 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:17,440 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:17,444 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:17,444 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:19,444 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:19,565 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:19,571 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:19,571 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:21,571 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:21,743 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:21,750 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:21,750 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:23,750 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:23,921 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:23,924 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:23,924 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:25,924 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:26,104 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:26,108 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:26,108 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:28,108 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:28,279 | test | WARNING | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-15 22:32:28,283 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:28,283 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:30,283 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:30,440 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:30,440 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:32,440 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:32,591 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:32,591 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:34,591 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:34,732 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:34,733 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:36,733 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:36,874 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:36,874 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:38,874 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:39,028 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:39,028 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:41,028 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:41,184 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:41,184 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:43,184 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:43,348 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:43,348 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:45,348 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:45,502 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:45,502 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:47,503 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:47,716 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:47,716 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:49,716 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:49,937 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:49,937 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:51,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:52,101 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:52,101 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:54,101 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:54,250 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:54,252 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:56,252 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:56,414 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:56,414 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:32:58,415 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:32:58,569 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:32:58,569 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:00,569 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:00,724 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:00,724 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:02,726 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:02,887 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:02,887 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:04,887 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:05,049 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:05,049 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:07,049 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:07,207 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:07,207 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:09,209 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:09,365 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:09,365 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:11,367 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:11,526 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:11,526 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:13,526 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:13,681 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:13,683 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:15,683 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:15,842 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:15,842 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:17,844 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:18,006 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:18,006 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:20,006 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:20,171 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:20,171 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:22,171 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:22,331 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:22,331 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:24,332 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:24,493 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:24,493 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:26,493 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:26,654 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:26,654 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:28,654 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:28,815 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:28,816 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:30,816 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:30,976 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:30,976 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:32,976 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:33,134 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:33,134 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:35,135 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:35,301 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:35,302 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:37,302 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:37,476 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:37,477 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:39,477 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:39,658 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:39,660 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:41,660 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:41,838 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:41,839 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:43,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:44,028 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:44,028 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-15 22:33:46,029 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:46,188 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 22:33:46,188 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.0819997787. 2021-12-15 22:33:46,190 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:33:46,193 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-15 22:33:46,325 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:33:46,615 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-15 22:33:46,680 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:33:46,680 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-15 22:33:46,694 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-15 22:33:46,759 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-15 22:33:46,835 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-15 22:33:46,852 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-15 22:33:46,865 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-15 22:33:46,882 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-15 22:33:46,898 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:47,065 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-15 22:33:47,065 | test | INFO | Thread-2 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.66 2021-12-15 22:33:49,079 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:49,578 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-15 22:33:49,578 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.38899993896. 2021-12-15 22:33:49,579 | test | DEBUG | Thread-2 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-15 22:33:49,582 | infra | DEBUG | Thread-2 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 22:33:49,645 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 22:33:49,841 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 22:33:49,898 | infra | DEBUG | Thread-2 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 22:33:49,900 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 22:33:49,911 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 22:33:49,923 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 22:33:49,937 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 22:33:49,953 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 22:33:49,966 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 22:33:49,977 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 22:33:49,990 | infra | DEBUG | Thread-2 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-15 22:33:50,141 | infra | DEBUG | Thread-2 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 22:33:50,142 | test | INFO | Thread-2 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.56299996376. 2021-12-15 22:33:50,145 | test | DEBUG | Thread-2 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-15 22:33:50,145 | test | DEBUG | Thread-2 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 8: Rebalance In_Out with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.66'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.67', '172.23.110.68'] Servers coming in : ['172.23.110.68', '172.23.110.67'] with services: ['kv', 'kv'] Servers going out : ['172.23.110.66'] NEW AVAILABLE nodes for cluster: ['172.23.110.66'] 2021-12-15 22:38:50,178 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.68,172.23.110.67]_OUT=[172.23.110.66]_1639636730.15 2021-12-15 22:38:50,180 | infra | INFO | pool-3-thread-30 | [task:start_task:90] Thread 'Rebalance_task_IN=[172.23.110.68,172.23.110.67]_OUT=[172.23.110.66]_1639636730.15' started 2021-12-15 22:38:50,184 | test | DEBUG | pool-3-thread-30 | [rest_client:add_node:1184] Adding remote node 172.23.110.68:8091 to cluster 172.23.110.64:8091 2021-12-15 22:38:50,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L518 2021-12-15 22:38:50,190 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-15 22:38:54,062 | test | DEBUG | pool-3-thread-30 | [rest_client:add_node:1184] Adding remote node 172.23.110.67:8091 to cluster 172.23.110.64:8091 2021-12-15 22:38:58,562 | test | DEBUG | pool-3-thread-30 | [task:start_rebalance:356] Removing node 172.23.110.66:8091 from cluster 2021-12-15 22:38:58,562 | test | DEBUG | pool-3-thread-30 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.66', 'user': 'Administrator'} 2021-12-15 22:38:58,614 | test | DEBUG | pool-3-thread-30 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 22:38:58,617 | test | INFO | pool-3-thread-30 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 19.6857334751 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 20.9441303253 | --- OUT ---> | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 16.7330844415 | Cluster node | | 172.23.110.68 | ['kv'] | | | <--- IN --- | | 172.23.110.67 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-15 22:38:58,625 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:38:58,625 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:38:58,638 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:38:58,641 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:38:58,641 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:08,651 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:08,651 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:08,667 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:08,668 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:08,670 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:18,681 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:18,681 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:18,698 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:18,700 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:18,700 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:28,711 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:28,713 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:28,727 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:28,730 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:28,730 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:38,743 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:38,743 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:38,760 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:38,763 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:38,763 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:48,776 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:48,776 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:48,792 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:48,795 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:48,795 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:39:50,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-15 22:39:50,200 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-15 22:39:58,806 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:39:58,806 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:39:58,823 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:39:58,825 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:39:58,826 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:08,838 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:40:08,838 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:40:08,854 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:08,855 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:08,855 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:18,869 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:40:18,869 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:40:18,885 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:18,888 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:18,888 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:20,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:40:20,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:40:20,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:28,900 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:40:28,901 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 22:40:28,917 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:28,918 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:28,918 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:30,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 22:40:30,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:40:30,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:38,934 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:40:38,934 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.09 2021-12-15 22:40:38,951 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:38,953 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:38,953 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:40,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:40:40,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:40:40,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:48,966 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:40:48,967 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.09 2021-12-15 22:40:48,996 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:48,997 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:48,997 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:50,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:40:50,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:40:50,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:40:59,009 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:40:59,009 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.09 2021-12-15 22:40:59,028 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:40:59,029 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:40:59,029 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:00,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.09 % 2021-12-15 22:41:00,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:00,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:09,043 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:09,043 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.48 2021-12-15 22:41:09,062 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:09,063 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:09,063 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:10,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:10,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:10,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:19,078 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:19,078 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.48 2021-12-15 22:41:19,098 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:19,099 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:19,099 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:20,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:20,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:20,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:29,111 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:29,112 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.48 2021-12-15 22:41:29,128 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:29,131 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:29,131 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:30,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:30,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:30,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:39,144 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:39,144 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.48 2021-12-15 22:41:39,161 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:39,163 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:39,163 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:40,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.48 % 2021-12-15 22:41:40,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:40,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:49,177 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:41:49,177 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-15 22:41:49,197 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:49,200 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:49,200 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:50,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:41:50,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:41:50,426 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:41:59,214 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:41:59,214 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-15 22:41:59,232 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:41:59,234 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:41:59,236 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:00,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:42:00,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:00,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:09,246 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:42:09,246 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-15 22:42:09,263 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:09,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:09,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:10,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-15 22:42:10,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:10,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:19,276 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.70 % 2021-12-15 22:42:19,276 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.7 2021-12-15 22:42:19,295 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:19,298 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:19,298 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:20,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.70 % 2021-12-15 22:42:20,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:20,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:29,309 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:29,309 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.79 2021-12-15 22:42:29,325 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:29,328 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:29,328 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:30,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:30,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:30,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:39,338 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:39,339 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.79 2021-12-15 22:42:39,355 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:39,358 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:39,358 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:40,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:40,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:40,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:49,368 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:49,369 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.79 2021-12-15 22:42:49,385 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:49,387 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:49,388 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:50,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.79 % 2021-12-15 22:42:50,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:42:50,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:42:59,400 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:42:59,400 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.89 2021-12-15 22:42:59,417 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:42:59,420 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:42:59,420 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:00,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:43:00,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:00,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:09,431 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:43:09,431 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.89 2021-12-15 22:43:09,451 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:09,453 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:09,453 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:10,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:43:10,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:10,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:19,466 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:43:19,467 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 0.89 2021-12-15 22:43:19,484 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:19,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:19,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:20,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.89 % 2021-12-15 22:43:20,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:20,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:29,500 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.11 % 2021-12-15 22:43:29,500 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.11 2021-12-15 22:43:29,519 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:29,522 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:29,522 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:30,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.11 % 2021-12-15 22:43:30,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:30,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:39,533 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.23 % 2021-12-15 22:43:39,533 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.23 2021-12-15 22:43:39,551 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:39,552 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:39,552 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:40,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.23 % 2021-12-15 22:43:40,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:40,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:49,565 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.36 % 2021-12-15 22:43:49,565 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.36 2021-12-15 22:43:49,582 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:49,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:49,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:50,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.36 % 2021-12-15 22:43:50,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:43:50,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:43:59,601 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.36 % 2021-12-15 22:43:59,601 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.36 2021-12-15 22:43:59,621 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:43:59,625 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:43:59,625 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:00,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:00,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:00,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:09,640 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:09,641 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.45 2021-12-15 22:44:09,657 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:09,661 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:09,661 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:10,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:10,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:10,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:19,674 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:19,674 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.45 2021-12-15 22:44:19,693 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:19,696 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:19,696 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:20,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:20,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:20,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:29,707 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:29,707 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.45 2021-12-15 22:44:29,724 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:29,726 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:29,726 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:30,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.45 % 2021-12-15 22:44:30,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:30,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:39,740 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:44:39,740 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.67 2021-12-15 22:44:39,759 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:39,763 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:39,763 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:40,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:44:40,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:40,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:49,775 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:44:49,775 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.67 2021-12-15 22:44:49,792 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:49,795 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:49,796 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:50,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:44:50,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:44:50,854 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:44:59,806 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:44:59,808 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.67 2021-12-15 22:44:59,825 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:44:59,828 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:44:59,828 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:00,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:45:00,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:00,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:09,838 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.67 % 2021-12-15 22:45:09,839 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.67 2021-12-15 22:45:09,857 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:45:09,859 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:45:09,859 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:10,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:10,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:10,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:19,871 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:19,871 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 22:45:19,888 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:45:19,891 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:45:19,891 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:20,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:20,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:20,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:29,903 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:29,903 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 22:45:29,920 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:45:29,921 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:45:29,921 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:30,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:30,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:30,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:39,934 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:39,934 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.76 2021-12-15 22:45:39,953 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:45:39,957 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:45:39,957 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:40,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-15 22:45:40,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:40,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:49,970 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.98 % 2021-12-15 22:45:49,970 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 1.98 2021-12-15 22:45:49,986 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:45:49,989 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:45:49,989 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:45:50,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.98 % 2021-12-15 22:45:50,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:45:50,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:00,000 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 22:46:00,000 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.11 2021-12-15 22:46:00,019 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:00,020 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:00,020 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:01,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 22:46:01,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:01,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:10,032 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 22:46:10,032 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.11 2021-12-15 22:46:10,052 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:10,053 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:10,053 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:11,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 22:46:11,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:11,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:20,068 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:20,068 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.33 2021-12-15 22:46:20,088 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:20,091 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:20,091 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:21,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:21,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:21,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:30,104 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:30,105 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.33 2021-12-15 22:46:30,132 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:30,137 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:30,137 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:31,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:31,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:31,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:40,148 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:40,148 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.33 2021-12-15 22:46:40,167 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:40,168 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:40,168 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:41,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:41,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:41,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:50,180 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:50,180 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.33 2021-12-15 22:46:50,200 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:46:50,203 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:46:50,203 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:46:51,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.33 % 2021-12-15 22:46:51,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:46:51,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:00,216 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 22:47:00,217 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.45 2021-12-15 22:47:00,236 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:00,240 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:00,240 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:01,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.45 % 2021-12-15 22:47:01,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:01,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:10,253 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.55 % 2021-12-15 22:47:10,253 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.55 2021-12-15 22:47:10,273 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:10,276 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:10,276 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:11,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.55 % 2021-12-15 22:47:11,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:11,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:20,288 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.55 % 2021-12-15 22:47:20,288 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.55 2021-12-15 22:47:20,305 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:20,306 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:20,308 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:21,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.55 % 2021-12-15 22:47:21,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:21,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:30,321 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:30,321 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.64 2021-12-15 22:47:30,339 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:30,342 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:30,342 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:31,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:31,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:31,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:40,357 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:40,358 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.64 2021-12-15 22:47:40,375 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:40,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:40,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:41,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:41,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:41,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:50,390 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:50,391 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.64 2021-12-15 22:47:50,408 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:47:50,411 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:47:50,411 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:47:51,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:47:51,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:47:51,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:00,424 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:48:00,424 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.64 2021-12-15 22:48:00,440 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:00,444 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:00,444 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:01,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.64 % 2021-12-15 22:48:01,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:01,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:10,454 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.77 % 2021-12-15 22:48:10,456 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.77 2021-12-15 22:48:10,473 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:10,474 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:10,474 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:11,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.77 % 2021-12-15 22:48:11,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:11,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:20,486 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.99 % 2021-12-15 22:48:20,486 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 2.99 2021-12-15 22:48:20,506 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:20,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:20,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:21,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.99 % 2021-12-15 22:48:21,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:21,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:30,526 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.11 % 2021-12-15 22:48:30,526 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.11 2021-12-15 22:48:30,545 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:30,548 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:30,548 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:31,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.11 % 2021-12-15 22:48:31,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:31,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:40,559 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:48:40,561 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 22:48:40,579 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:40,582 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:40,584 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:41,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:48:41,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:41,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:50,605 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:48:50,607 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 22:48:50,628 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:48:50,631 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:48:50,631 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:48:51,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:48:51,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:48:51,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:00,642 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:49:00,644 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 22:49:00,663 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:00,665 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:00,665 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:01,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:49:01,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:01,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:10,680 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:49:10,680 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-15 22:49:10,700 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:10,703 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:10,703 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:11,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-15 22:49:11,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:11,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:20,713 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.33 % 2021-12-15 22:49:20,714 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.33 2021-12-15 22:49:20,730 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:20,734 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:20,734 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:21,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.33 % 2021-12-15 22:49:21,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:21,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:30,746 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.33 % 2021-12-15 22:49:30,746 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.33 2021-12-15 22:49:30,763 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:30,765 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:30,765 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:31,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.33 % 2021-12-15 22:49:31,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:31,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:40,776 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.42 % 2021-12-15 22:49:40,776 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.42 2021-12-15 22:49:40,792 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:40,796 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:40,796 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:41,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.42 % 2021-12-15 22:49:41,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:41,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:50,811 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:49:50,811 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 22:49:50,831 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:49:50,834 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:49:50,834 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:49:51,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:49:51,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:49:51,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:00,845 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:00,845 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 22:50:00,865 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:00,868 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:00,868 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:01,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:01,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:01,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:10,891 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:10,891 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 22:50:10,910 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:10,911 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:10,913 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:11,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:11,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:11,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:20,924 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:20,924 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.52 2021-12-15 22:50:20,940 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:20,943 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:20,943 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:21,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.52 % 2021-12-15 22:50:21,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:21,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:30,954 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.77 % 2021-12-15 22:50:30,954 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.77 2021-12-15 22:50:30,970 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:30,973 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:30,973 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:31,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.77 % 2021-12-15 22:50:31,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:31,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:40,983 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.77 % 2021-12-15 22:50:40,984 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.77 2021-12-15 22:50:41,003 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:41,007 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:41,007 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:41,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.77 % 2021-12-15 22:50:41,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:41,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:51,022 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.86 % 2021-12-15 22:50:51,023 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 3.86 2021-12-15 22:50:51,042 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:50:51,043 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:50:51,045 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:50:51,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.86 % 2021-12-15 22:50:51,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:50:51,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:01,055 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:01,055 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.08 2021-12-15 22:51:01,075 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:01,078 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:01,078 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:01,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:01,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:01,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:11,089 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:11,089 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.08 2021-12-15 22:51:11,108 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:11,109 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:11,109 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:11,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:11,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:11,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:21,121 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:21,122 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.08 2021-12-15 22:51:21,140 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:21,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:21,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:21,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:21,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:21,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:31,157 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:31,157 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.08 2021-12-15 22:51:31,174 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:31,177 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:31,177 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:31,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.08 % 2021-12-15 22:51:31,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:31,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:41,188 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.21 % 2021-12-15 22:51:41,188 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.21 2021-12-15 22:51:41,209 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:41,211 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:41,211 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:41,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.21 % 2021-12-15 22:51:41,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:41,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:51,224 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.21 % 2021-12-15 22:51:51,224 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.21 2021-12-15 22:51:51,243 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:51:51,246 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:51:51,246 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:51:51,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.21 % 2021-12-15 22:51:51,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:51:51,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:01,260 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:01,262 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.39 2021-12-15 22:52:01,296 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:01,299 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:01,299 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:01,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:01,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:01,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:11,311 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:11,311 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.39 2021-12-15 22:52:11,328 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:11,331 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:11,331 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:11,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:11,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:11,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:21,342 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:21,342 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.39 2021-12-15 22:52:21,358 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:21,361 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:21,361 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:21,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:21,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:21,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:31,371 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:31,372 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.39 2021-12-15 22:52:31,390 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:31,392 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:31,392 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:31,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:31,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:31,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:41,404 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:41,404 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.39 2021-12-15 22:52:41,421 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:41,423 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:41,424 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:41,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.39 % 2021-12-15 22:52:41,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:41,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:51,437 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.52 % 2021-12-15 22:52:51,437 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.52 2021-12-15 22:52:51,453 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:52:51,456 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:52:51,456 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:52:51,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.52 % 2021-12-15 22:52:51,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:52:51,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:01,467 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.52 % 2021-12-15 22:53:01,467 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.52 2021-12-15 22:53:01,484 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:01,486 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:01,487 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:01,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.64 % 2021-12-15 22:53:01,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:01,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:11,499 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.83 % 2021-12-15 22:53:11,499 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.83 2021-12-15 22:53:11,516 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:11,519 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:11,519 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:11,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.83 % 2021-12-15 22:53:11,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:11,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:21,529 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:21,530 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.96 2021-12-15 22:53:21,548 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:21,549 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:21,551 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:21,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:21,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:21,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:31,562 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:31,562 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.96 2021-12-15 22:53:31,578 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:31,582 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:31,582 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:31,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:31,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:31,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:41,594 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:41,594 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 4.96 2021-12-15 22:53:41,612 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:41,615 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:41,615 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:42,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.96 % 2021-12-15 22:53:42,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:42,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:51,631 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:53:51,631 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.08 2021-12-15 22:53:51,651 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:53:51,654 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:53:51,654 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:53:52,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:53:52,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:53:52,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:01,667 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:54:01,667 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.08 2021-12-15 22:54:01,697 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:01,700 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:01,700 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:02,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:54:02,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:02,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:11,710 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:54:11,711 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.08 2021-12-15 22:54:11,729 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:11,730 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:11,732 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:12,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.08 % 2021-12-15 22:54:12,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:12,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:21,743 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:21,743 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 22:54:21,759 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:21,760 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:21,760 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:22,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:22,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:22,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:31,772 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:31,773 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 22:54:31,789 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:31,792 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:31,792 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:32,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:32,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:32,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:41,802 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:41,802 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 22:54:41,819 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:41,822 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:41,822 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:42,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:42,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:42,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:51,834 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:51,834 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.27 2021-12-15 22:54:51,851 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:54:51,854 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:54:51,854 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:54:52,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-15 22:54:52,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:54:52,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:01,865 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.40 % 2021-12-15 22:55:01,865 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.4 2021-12-15 22:55:01,881 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:01,884 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:01,884 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:02,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.40 % 2021-12-15 22:55:02,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:02,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:11,894 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.40 % 2021-12-15 22:55:11,894 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.4 2021-12-15 22:55:11,921 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:11,924 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:11,924 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:12,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.40 % 2021-12-15 22:55:12,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:12,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:21,934 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.52 % 2021-12-15 22:55:21,936 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.52 2021-12-15 22:55:21,951 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:21,953 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:21,953 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:22,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.52 % 2021-12-15 22:55:22,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:22,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:31,964 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.52 % 2021-12-15 22:55:31,964 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.52 2021-12-15 22:55:31,980 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:31,983 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:31,983 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:32,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.52 % 2021-12-15 22:55:32,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:32,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:41,994 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:55:41,994 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.83 2021-12-15 22:55:42,012 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:42,013 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:42,013 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:42,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:55:42,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:42,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:52,025 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:55:52,026 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.83 2021-12-15 22:55:52,043 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:55:52,046 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:55:52,046 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:55:52,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:55:52,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:55:52,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:02,058 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:56:02,059 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.83 2021-12-15 22:56:02,075 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:02,076 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:02,078 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:02,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.83 % 2021-12-15 22:56:02,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:02,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:12,088 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:12,089 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.96 2021-12-15 22:56:12,105 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:12,108 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:12,108 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:12,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:12,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:12,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:22,121 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:22,121 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.96 2021-12-15 22:56:22,138 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:22,141 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:22,141 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:22,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:22,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:22,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:32,153 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:32,153 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 5.96 2021-12-15 22:56:32,168 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:32,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:32,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:32,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.96 % 2021-12-15 22:56:32,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:32,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:42,184 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.05 % 2021-12-15 22:56:42,184 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.05 2021-12-15 22:56:42,201 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:42,204 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:42,204 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:42,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.05 % 2021-12-15 22:56:42,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:42,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:52,216 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:56:52,217 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.15 2021-12-15 22:56:52,232 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:56:52,234 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:56:52,234 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:56:52,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:56:52,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:56:52,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:02,246 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:57:02,247 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.15 2021-12-15 22:57:02,265 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:02,266 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:02,267 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:02,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:57:02,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:02,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:12,279 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:57:12,279 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.15 2021-12-15 22:57:12,296 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:12,299 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:12,299 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:12,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.15 % 2021-12-15 22:57:12,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:12,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:22,311 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.27 % 2021-12-15 22:57:22,312 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.27 2021-12-15 22:57:22,329 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:22,332 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:22,332 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:22,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.27 % 2021-12-15 22:57:22,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:22,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:32,345 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.40 % 2021-12-15 22:57:32,345 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.4 2021-12-15 22:57:32,362 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:32,365 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:32,365 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:32,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.40 % 2021-12-15 22:57:32,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:32,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:42,378 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.40 % 2021-12-15 22:57:42,378 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.4 2021-12-15 22:57:42,398 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:42,400 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:42,401 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:42,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.40 % 2021-12-15 22:57:42,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:42,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:52,411 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.62 % 2021-12-15 22:57:52,411 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.62 2021-12-15 22:57:52,427 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:57:52,430 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:57:52,430 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:57:52,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.62 % 2021-12-15 22:57:52,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:57:52,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:02,441 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:02,441 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.71 2021-12-15 22:58:02,457 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:02,460 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:02,460 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:02,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:02,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:02,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:12,470 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:12,470 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.71 2021-12-15 22:58:12,490 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:12,492 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:12,492 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:12,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:12,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:12,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:22,503 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:22,505 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.71 2021-12-15 22:58:22,519 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:22,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:22,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:22,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.71 % 2021-12-15 22:58:22,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:22,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:32,533 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 22:58:32,533 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 22:58:32,551 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:32,553 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:32,553 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:32,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 22:58:32,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:32,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:42,565 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 22:58:42,565 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.84 2021-12-15 22:58:42,582 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:42,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:42,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:42,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.84 % 2021-12-15 22:58:42,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:42,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:52,596 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:58:52,596 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.93 2021-12-15 22:58:52,614 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:58:52,617 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:58:52,617 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:58:52,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:58:52,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:58:52,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:02,628 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:59:02,628 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.93 2021-12-15 22:59:02,647 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:02,648 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:02,650 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:02,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:59:02,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:02,677 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:12,660 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:59:12,661 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 6.93 2021-12-15 22:59:12,677 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:12,678 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:12,680 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:12,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.93 % 2021-12-15 22:59:12,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:12,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:22,690 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 22:59:22,690 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.02 2021-12-15 22:59:22,709 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:22,711 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:22,711 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:22,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 22:59:22,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:22,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:32,723 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 22:59:32,723 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.02 2021-12-15 22:59:32,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-15 22:59:32,742 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:32,746 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:32,746 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:32,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:32,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:42,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.15 % 2021-12-15 22:59:42,760 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.15 % 2021-12-15 22:59:42,760 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.15 2021-12-15 22:59:42,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:42,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:42,776 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:42,779 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:42,779 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:52,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.27 % 2021-12-15 22:59:52,789 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.27 % 2021-12-15 22:59:52,789 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.27 2021-12-15 22:59:52,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 22:59:52,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 22:59:52,806 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 22:59:52,809 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 22:59:52,809 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:02,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.37 % 2021-12-15 23:00:02,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:02,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:02,819 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.37 % 2021-12-15 23:00:02,819 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.37 2021-12-15 23:00:02,842 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:02,846 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:02,846 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:12,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-15 23:00:12,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:12,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:12,874 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-15 23:00:12,874 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.49 2021-12-15 23:00:12,892 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:12,895 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:12,895 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:22,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-15 23:00:22,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:22,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:22,907 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-15 23:00:22,907 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.49 2021-12-15 23:00:22,924 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:22,927 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:22,927 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:32,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:00:32,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:32,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:32,938 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:00:32,940 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.59 2021-12-15 23:00:32,959 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:32,960 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:32,961 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:42,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:00:42,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:42,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:42,973 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:00:42,973 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.59 2021-12-15 23:00:42,996 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:43,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:43,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:52,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-15 23:00:52,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:00:52,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:00:53,016 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-15 23:00:53,016 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.71 2021-12-15 23:00:53,036 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:00:53,042 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:00:53,042 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:02,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-15 23:01:02,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:02,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:03,055 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-15 23:01:03,055 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.71 2021-12-15 23:01:03,075 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:03,078 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:03,078 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:12,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:12,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:12,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:13,092 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:13,092 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.81 2021-12-15 23:01:13,109 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:13,112 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:13,112 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:22,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:23,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:23,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:23,122 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:23,122 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.81 2021-12-15 23:01:23,141 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:23,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:23,144 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:33,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:33,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:33,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:33,157 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.81 % 2021-12-15 23:01:33,157 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.81 2021-12-15 23:01:33,176 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:33,178 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:33,178 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:43,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.90 % 2021-12-15 23:01:43,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:43,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:43,191 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.90 % 2021-12-15 23:01:43,191 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.9 2021-12-15 23:01:43,209 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:43,210 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:43,211 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:53,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.90 % 2021-12-15 23:01:53,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:01:53,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:01:53,221 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.90 % 2021-12-15 23:01:53,223 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 7.9 2021-12-15 23:01:53,240 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:01:53,243 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:01:53,243 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:03,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 23:02:03,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:03,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:03,253 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-15 23:02:03,253 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-15 23:02:03,269 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:03,270 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:03,272 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:13,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.12 % 2021-12-15 23:02:13,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:13,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:13,282 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.12 % 2021-12-15 23:02:13,282 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.12 2021-12-15 23:02:13,301 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:13,303 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:13,303 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:23,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.24 % 2021-12-15 23:02:23,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:23,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:23,315 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.24 % 2021-12-15 23:02:23,315 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.24 2021-12-15 23:02:23,332 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:23,335 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:23,335 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:33,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.24 % 2021-12-15 23:02:33,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:33,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:33,349 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.24 % 2021-12-15 23:02:33,349 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.24 2021-12-15 23:02:33,367 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:33,369 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:33,369 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:43,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.37 % 2021-12-15 23:02:43,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:43,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:43,384 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.37 % 2021-12-15 23:02:43,384 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.37 2021-12-15 23:02:43,404 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:43,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:43,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:53,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.37 % 2021-12-15 23:02:53,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:02:53,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:02:53,418 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.37 % 2021-12-15 23:02:53,418 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.37 2021-12-15 23:02:53,436 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:02:53,437 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:02:53,437 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:03,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.46 % 2021-12-15 23:03:03,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:03,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:03,450 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.46 % 2021-12-15 23:03:03,450 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.46 2021-12-15 23:03:03,467 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:03,470 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:03,470 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:13,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.46 % 2021-12-15 23:03:13,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:13,247 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:13,480 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.46 % 2021-12-15 23:03:13,480 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.46 2021-12-15 23:03:13,497 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:13,500 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:13,500 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:23,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:23,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:23,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:23,512 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:23,512 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-15 23:03:23,529 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:23,532 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:23,532 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:33,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:33,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:33,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:33,546 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:33,546 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-15 23:03:33,566 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:33,569 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:33,569 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:43,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:43,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:43,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:43,581 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:43,581 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-15 23:03:43,598 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:43,601 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:43,601 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:53,344 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:53,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:03:53,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:03:53,611 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:03:53,612 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-15 23:03:53,630 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:03:53,631 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:03:53,631 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:03,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:04:03,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:03,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:03,642 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-15 23:04:03,644 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-15 23:04:03,661 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:03,664 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:03,664 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:13,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-15 23:04:13,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:13,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:13,674 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-15 23:04:13,674 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.78 2021-12-15 23:04:13,691 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:13,694 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:13,694 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:23,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-15 23:04:23,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:23,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:23,704 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-15 23:04:23,704 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 8.78 2021-12-15 23:04:23,721 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:23,724 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:23,724 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:33,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.87 % 2021-12-15 23:04:33,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:33,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:33,736 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.00 % 2021-12-15 23:04:33,736 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.0 2021-12-15 23:04:33,753 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:33,756 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:33,756 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:43,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.00 % 2021-12-15 23:04:43,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:43,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:43,766 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.00 % 2021-12-15 23:04:43,766 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.0 2021-12-15 23:04:43,783 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:43,786 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:43,786 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:53,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.12 % 2021-12-15 23:04:53,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:04:53,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:04:53,796 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.12 % 2021-12-15 23:04:53,796 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.12 2021-12-15 23:04:53,812 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:04:53,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:04:53,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:03,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.25 % 2021-12-15 23:05:03,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:03,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:03,825 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.25 % 2021-12-15 23:05:03,826 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.25 2021-12-15 23:05:03,844 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:03,845 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:03,845 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:13,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.25 % 2021-12-15 23:05:13,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:13,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:13,858 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.25 % 2021-12-15 23:05:13,858 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.25 2021-12-15 23:05:13,878 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:13,881 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:13,881 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:23,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.34 % 2021-12-15 23:05:23,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:23,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:23,891 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.34 % 2021-12-15 23:05:23,891 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.34 2021-12-15 23:05:23,911 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:23,915 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:23,915 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:33,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.43 % 2021-12-15 23:05:33,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:33,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:33,936 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.43 % 2021-12-15 23:05:33,936 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.43 2021-12-15 23:05:33,954 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:33,957 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:33,957 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:43,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:05:43,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:43,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:43,967 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:05:43,967 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.56 2021-12-15 23:05:43,986 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:43,989 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:43,989 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:53,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:05:53,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:05:53,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:05:54,000 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:05:54,000 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.56 2021-12-15 23:05:54,017 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:05:54,019 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:05:54,019 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:03,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:03,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:03,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:04,032 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:04,032 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.56 2021-12-15 23:06:04,049 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:04,052 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:04,052 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:13,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:13,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:13,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:14,062 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:14,063 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.56 2021-12-15 23:06:14,079 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:14,081 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:14,082 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:23,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:23,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:23,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:24,094 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-15 23:06:24,095 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.56 2021-12-15 23:06:24,112 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:24,115 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:24,115 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:33,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.65 % 2021-12-15 23:06:33,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:33,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:34,125 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.65 % 2021-12-15 23:06:34,125 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.65 2021-12-15 23:06:34,150 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:34,154 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:34,154 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:43,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.65 % 2021-12-15 23:06:43,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:43,719 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:44,164 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.65 % 2021-12-15 23:06:44,164 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.65 2021-12-15 23:06:44,180 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:44,183 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:44,183 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:53,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.75 % 2021-12-15 23:06:53,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:06:53,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:06:54,193 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.75 % 2021-12-15 23:06:54,193 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.75 2021-12-15 23:06:54,209 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:06:54,210 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:06:54,210 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:03,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 23:07:03,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:03,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:04,220 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.87 % 2021-12-15 23:07:04,221 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 9.87 2021-12-15 23:07:04,239 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:04,242 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:04,242 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:13,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-15 23:07:13,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:13,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:14,253 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-15 23:07:14,253 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.0 2021-12-15 23:07:14,272 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:14,275 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:14,275 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:23,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-15 23:07:23,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:23,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:24,286 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-15 23:07:24,286 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.0 2021-12-15 23:07:24,305 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:24,308 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:24,309 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:33,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.12 % 2021-12-15 23:07:33,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:33,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:34,319 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.12 % 2021-12-15 23:07:34,319 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.12 2021-12-15 23:07:34,335 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:34,338 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:34,338 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:43,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.22 % 2021-12-15 23:07:43,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:43,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:44,352 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.22 % 2021-12-15 23:07:44,352 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.22 2021-12-15 23:07:44,371 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:44,375 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:44,375 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:53,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:07:53,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:07:53,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:07:54,388 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:07:54,388 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-15 23:07:54,404 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:07:54,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:07:54,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:03,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:08:03,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:03,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:04,418 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:08:04,420 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-15 23:08:04,437 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:04,440 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:04,440 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:13,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:08:13,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:13,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:14,454 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:08:14,456 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-15 23:08:14,480 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:14,483 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:14,484 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:23,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-15 23:08:23,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:23,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:24,500 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:24,500 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.44 2021-12-15 23:08:24,522 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:24,526 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:24,526 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:33,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:33,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:33,974 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:34,539 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:34,539 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.44 2021-12-15 23:08:34,558 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:34,561 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:34,561 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:43,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:44,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:44,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:44,572 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:44,572 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.44 2021-12-15 23:08:44,589 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:44,592 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:44,592 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:54,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:54,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:08:54,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:08:54,608 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.44 % 2021-12-15 23:08:54,608 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.44 2021-12-15 23:08:54,631 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:08:54,634 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:08:54,634 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:04,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.62 % 2021-12-15 23:09:04,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:04,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:04,645 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.62 % 2021-12-15 23:09:04,645 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.62 2021-12-15 23:09:04,663 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:04,664 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:04,664 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:14,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.62 % 2021-12-15 23:09:14,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:14,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:14,678 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.62 % 2021-12-15 23:09:14,678 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.62 2021-12-15 23:09:14,696 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:14,698 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:14,698 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:24,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 23:09:24,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:24,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:24,711 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 23:09:24,713 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.75 2021-12-15 23:09:24,730 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:24,733 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:24,733 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:34,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 23:09:34,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:34,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:34,744 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.75 % 2021-12-15 23:09:34,744 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 10.75 2021-12-15 23:09:34,766 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:34,769 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:34,769 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:44,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:09:44,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:44,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:44,782 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:09:44,782 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.0 2021-12-15 23:09:44,798 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:44,801 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:44,801 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:54,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:09:54,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:09:54,155 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:09:54,813 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:09:54,813 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.0 2021-12-15 23:09:54,829 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:09:54,832 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:09:54,832 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:04,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:10:04,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:04,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:04,845 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.00 % 2021-12-15 23:10:04,845 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.0 2021-12-15 23:10:04,864 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:04,865 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:04,865 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:14,193 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.09 % 2021-12-15 23:10:14,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:14,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:14,878 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.09 % 2021-12-15 23:10:14,880 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.09 2021-12-15 23:10:14,898 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:14,901 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:14,901 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:24,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.19 % 2021-12-15 23:10:24,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:24,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:24,914 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.19 % 2021-12-15 23:10:24,915 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.19 2021-12-15 23:10:24,934 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:24,937 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:24,937 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:34,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.19 % 2021-12-15 23:10:34,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:34,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:34,950 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.19 % 2021-12-15 23:10:34,951 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.19 2021-12-15 23:10:34,969 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:34,971 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:34,973 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:44,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:10:44,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:44,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:44,989 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:10:44,990 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.31 2021-12-15 23:10:45,006 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:45,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:45,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:54,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:10:54,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:10:54,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:10:55,022 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:10:55,022 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.31 2021-12-15 23:10:55,038 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:10:55,039 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:10:55,040 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:04,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:11:04,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:04,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:05,052 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:11:05,052 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.31 2021-12-15 23:11:05,069 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:05,072 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:05,072 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:14,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:11:14,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:14,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:15,085 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.31 % 2021-12-15 23:11:15,085 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.31 2021-12-15 23:11:15,119 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:15,122 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:15,122 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:24,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.41 % 2021-12-15 23:11:24,344 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:24,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:25,142 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.41 % 2021-12-15 23:11:25,142 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.41 2021-12-15 23:11:25,161 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:25,164 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:25,164 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:34,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.50 % 2021-12-15 23:11:34,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:34,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:35,177 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.50 % 2021-12-15 23:11:35,177 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.5 2021-12-15 23:11:35,191 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:35,194 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:35,194 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:44,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.63 % 2021-12-15 23:11:44,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:44,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:45,206 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.63 % 2021-12-15 23:11:45,206 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.63 2021-12-15 23:11:45,224 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:45,226 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:45,227 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:54,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.63 % 2021-12-15 23:11:54,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:11:54,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:11:55,239 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.63 % 2021-12-15 23:11:55,239 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.63 2021-12-15 23:11:55,256 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:11:55,259 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:11:55,259 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:04,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:04,418 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:04,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:05,272 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:05,272 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.88 2021-12-15 23:12:05,288 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:05,289 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:05,290 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:14,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:14,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:14,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:15,302 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:15,302 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.88 2021-12-15 23:12:15,319 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:15,322 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:15,322 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:24,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:24,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:24,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:25,332 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.88 % 2021-12-15 23:12:25,332 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.88 2021-12-15 23:12:25,349 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:25,351 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:25,352 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:34,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 23:12:34,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:34,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:35,364 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.97 % 2021-12-15 23:12:35,364 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 11.97 2021-12-15 23:12:35,380 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:35,381 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:35,381 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:44,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:12:44,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:44,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:45,392 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:12:45,394 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.06 2021-12-15 23:12:45,410 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:45,411 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:45,411 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:54,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:12:54,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:12:54,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:12:55,424 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:12:55,424 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.06 2021-12-15 23:12:55,443 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:12:55,444 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:12:55,444 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:04,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:13:04,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:04,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:05,457 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:13:05,457 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.06 2021-12-15 23:13:05,477 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:05,480 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:05,480 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:14,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:13:14,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:14,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:15,493 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.06 % 2021-12-15 23:13:15,493 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.06 2021-12-15 23:13:15,513 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:15,516 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:15,516 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:24,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-15 23:13:24,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:24,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:25,526 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-15 23:13:25,528 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-15 23:13:25,543 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:25,546 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:25,546 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:34,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-15 23:13:34,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:34,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:35,559 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-15 23:13:35,559 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-15 23:13:35,575 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:35,578 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:35,578 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:44,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 23:13:44,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:44,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:45,589 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.28 % 2021-12-15 23:13:45,591 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.28 2021-12-15 23:13:45,608 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:45,609 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:45,609 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:54,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 23:13:54,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:13:54,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:13:55,622 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 23:13:55,622 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.5 2021-12-15 23:13:55,641 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:13:55,644 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:13:55,644 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:04,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 23:14:04,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:04,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:05,654 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.50 % 2021-12-15 23:14:05,654 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.5 2021-12-15 23:14:05,673 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:05,676 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:05,677 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:14,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 23:14:14,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:14,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:15,690 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 23:14:15,690 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 23:14:15,707 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:15,710 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:15,710 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:24,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 23:14:24,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:24,703 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:25,724 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.63 % 2021-12-15 23:14:25,724 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.63 2021-12-15 23:14:25,743 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:25,744 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:25,744 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:34,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.75 % 2021-12-15 23:14:34,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:34,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:35,756 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.75 % 2021-12-15 23:14:35,756 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.75 2021-12-15 23:14:35,773 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:35,776 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:35,776 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:44,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.85 % 2021-12-15 23:14:44,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:44,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:45,788 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.85 % 2021-12-15 23:14:45,789 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.85 2021-12-15 23:14:45,808 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:45,811 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:45,812 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:54,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.85 % 2021-12-15 23:14:54,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:14:54,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:14:55,826 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.85 % 2021-12-15 23:14:55,826 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.85 2021-12-15 23:14:55,845 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:14:55,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:14:55,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:04,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:04,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:04,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:05,861 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:05,861 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.94 2021-12-15 23:15:05,881 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:05,884 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:05,884 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:14,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:14,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:14,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:15,901 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:15,901 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.94 2021-12-15 23:15:15,921 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:15,924 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:15,924 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:24,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:24,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:24,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:25,941 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:25,943 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.94 2021-12-15 23:15:25,960 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:25,963 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:25,963 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:34,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:34,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:34,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:35,976 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.94 % 2021-12-15 23:15:35,976 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 12.94 2021-12-15 23:15:35,996 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:36,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:36,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:44,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-15 23:15:44,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:44,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:46,013 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-15 23:15:46,015 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.07 2021-12-15 23:15:46,032 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:46,036 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:46,036 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:54,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.19 % 2021-12-15 23:15:54,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:15:54,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:15:56,049 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-15 23:15:56,051 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.28 2021-12-15 23:15:56,068 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:15:56,071 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:15:56,071 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:04,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.41 % 2021-12-15 23:16:04,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:04,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:06,088 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:06,088 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.5 2021-12-15 23:16:06,107 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:06,109 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:06,109 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:14,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:14,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:14,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:16,122 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:16,122 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.5 2021-12-15 23:16:16,141 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:16,142 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:16,142 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:24,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:24,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:24,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:26,155 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:26,155 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.5 2021-12-15 23:16:26,173 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:26,174 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:26,174 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:34,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:34,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:34,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:36,200 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:36,200 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.5 2021-12-15 23:16:36,219 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:36,221 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:36,221 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:44,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:45,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:45,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:46,233 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.50 % 2021-12-15 23:16:46,234 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.5 2021-12-15 23:16:46,253 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:46,256 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:46,256 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:55,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.63 % 2021-12-15 23:16:55,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:16:55,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:16:56,267 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.63 % 2021-12-15 23:16:56,269 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.63 2021-12-15 23:16:56,285 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:16:56,286 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:16:56,286 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:05,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.63 % 2021-12-15 23:17:05,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:05,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:06,299 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.72 % 2021-12-15 23:17:06,299 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.72 2021-12-15 23:17:06,315 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:06,316 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:06,316 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:15,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:15,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:15,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:16,328 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:16,328 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.82 2021-12-15 23:17:16,345 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:16,346 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:16,346 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:25,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:25,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:25,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:26,358 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:26,358 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.82 2021-12-15 23:17:26,375 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:26,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:26,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:35,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:35,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:35,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:36,388 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.82 % 2021-12-15 23:17:36,390 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.82 2021-12-15 23:17:36,405 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:36,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:36,407 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:45,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.94 % 2021-12-15 23:17:45,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:45,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:46,424 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.94 % 2021-12-15 23:17:46,424 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 13.94 2021-12-15 23:17:46,444 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:46,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:46,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:55,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 23:17:55,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:17:55,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:17:56,457 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.07 % 2021-12-15 23:17:56,459 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.07 2021-12-15 23:17:56,473 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:17:56,476 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:17:56,476 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:05,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.16 % 2021-12-15 23:18:05,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:05,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:06,489 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.16 % 2021-12-15 23:18:06,490 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.16 2021-12-15 23:18:06,507 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:06,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:06,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:15,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:15,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:15,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:16,520 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:16,520 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:18:16,539 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:16,542 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:16,542 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:25,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:25,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:25,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:26,553 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:26,553 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:18:26,575 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:26,576 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:26,576 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:35,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:35,252 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:35,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:36,588 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:36,589 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:18:36,608 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:36,609 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:36,609 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:45,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:45,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:45,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:46,622 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:46,622 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:18:46,640 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:46,642 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:46,642 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:55,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:55,293 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:18:55,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:18:56,654 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:18:56,655 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:18:56,674 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:18:56,677 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:18:56,677 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:05,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:19:05,313 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:05,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:06,691 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.35 % 2021-12-15 23:19:06,691 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.35 2021-12-15 23:19:06,710 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:06,713 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:06,713 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:15,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.57 % 2021-12-15 23:19:15,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:15,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:16,726 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 23:19:16,726 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 23:19:16,743 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:16,746 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:16,746 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:25,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 23:19:25,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:25,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:26,759 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.66 % 2021-12-15 23:19:26,759 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.66 2021-12-15 23:19:26,778 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:26,779 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:26,779 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:35,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.76 % 2021-12-15 23:19:35,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:35,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:36,792 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.76 % 2021-12-15 23:19:36,793 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 14.76 2021-12-15 23:19:36,816 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:36,819 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:36,819 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:45,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.99 % 2021-12-15 23:19:45,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:45,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:46,834 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.12 % 2021-12-15 23:19:46,834 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.12 2021-12-15 23:19:46,852 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:46,857 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:46,857 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:55,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.12 % 2021-12-15 23:19:55,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:19:55,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:19:56,871 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.12 % 2021-12-15 23:19:56,871 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.12 2021-12-15 23:19:56,892 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:19:56,895 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:19:56,897 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:05,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.12 % 2021-12-15 23:20:05,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:05,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:06,910 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.12 % 2021-12-15 23:20:06,910 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.12 2021-12-15 23:20:06,928 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:06,931 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:06,933 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:15,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.25 % 2021-12-15 23:20:15,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:15,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:16,946 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.25 % 2021-12-15 23:20:16,947 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.25 2021-12-15 23:20:16,964 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:16,967 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:16,967 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:25,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.34 % 2021-12-15 23:20:25,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:25,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:26,980 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.34 % 2021-12-15 23:20:26,980 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.34 2021-12-15 23:20:26,997 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:27,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:27,000 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:35,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:35,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:35,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:37,015 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:37,015 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.44 2021-12-15 23:20:37,032 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:37,035 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:37,035 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:45,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:45,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:45,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:47,046 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:47,046 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.44 2021-12-15 23:20:47,066 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:47,069 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:47,069 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:55,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:55,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:20:55,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:20:57,082 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:20:57,082 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.44 2021-12-15 23:20:57,102 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:20:57,105 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:20:57,105 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:05,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:21:05,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:05,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:07,121 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.44 % 2021-12-15 23:21:07,121 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.44 2021-12-15 23:21:07,140 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:07,142 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:07,142 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:15,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-15 23:21:15,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:15,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:17,171 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-15 23:21:17,171 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-15 23:21:17,194 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:17,198 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:17,198 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:25,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.81 % 2021-12-15 23:21:25,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:25,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:27,210 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.95 % 2021-12-15 23:21:27,210 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 15.95 2021-12-15 23:21:27,230 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:27,233 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:27,233 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:35,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.18 % 2021-12-15 23:21:35,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:35,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:37,246 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.18 % 2021-12-15 23:21:37,246 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.18 2021-12-15 23:21:37,262 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:37,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:37,265 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:45,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.18 % 2021-12-15 23:21:45,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:45,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:47,276 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.18 % 2021-12-15 23:21:47,278 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.18 2021-12-15 23:21:47,295 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:47,296 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:47,296 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:55,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.18 % 2021-12-15 23:21:55,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:21:55,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:21:57,309 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:21:57,309 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.27 2021-12-15 23:21:57,325 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:21:57,328 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:21:57,328 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:05,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:05,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:05,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:07,339 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:07,339 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.27 2021-12-15 23:22:07,355 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:07,358 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:07,358 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:15,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:15,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:15,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:17,371 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:17,371 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.27 2021-12-15 23:22:17,390 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:17,391 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:17,392 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:25,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:25,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:25,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:27,405 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.27 % 2021-12-15 23:22:27,405 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.27 2021-12-15 23:22:27,424 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:27,427 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:27,427 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:35,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.37 % 2021-12-15 23:22:35,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:35,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:37,440 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.49 % 2021-12-15 23:22:37,440 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.49 2021-12-15 23:22:37,457 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:37,460 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:37,460 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:45,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.49 % 2021-12-15 23:22:45,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:45,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:47,473 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.49 % 2021-12-15 23:22:47,473 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.49 2021-12-15 23:22:47,490 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:47,492 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:47,492 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:55,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.62 % 2021-12-15 23:22:55,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:22:55,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:22:57,505 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.62 % 2021-12-15 23:22:57,505 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.62 2021-12-15 23:22:57,520 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:22:57,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:22:57,523 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:05,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.62 % 2021-12-15 23:23:05,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:05,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:07,533 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.62 % 2021-12-15 23:23:07,533 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 16.62 2021-12-15 23:23:07,549 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:07,551 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:07,552 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:15,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.02 % 2021-12-15 23:23:15,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:15,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:17,563 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.02 % 2021-12-15 23:23:17,563 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.02 2021-12-15 23:23:17,582 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:17,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:17,585 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:25,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-15 23:23:25,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:25,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:27,598 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-15 23:23:27,598 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.11 2021-12-15 23:23:27,615 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:27,618 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:27,618 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:35,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-15 23:23:35,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:35,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:37,631 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.21 % 2021-12-15 23:23:37,631 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.21 2021-12-15 23:23:37,650 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:37,651 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:37,651 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:45,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.21 % 2021-12-15 23:23:45,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:45,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:47,664 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:23:47,664 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:23:47,680 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:47,681 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:47,681 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:55,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:23:55,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:23:55,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:23:57,694 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:23:57,696 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:23:57,713 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:23:57,717 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:23:57,717 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:05,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:05,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:05,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:07,730 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:07,730 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:24:07,746 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:07,749 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:07,749 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:15,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:15,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:15,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:17,763 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:17,763 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:24:17,782 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:17,783 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:17,783 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:25,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:25,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:25,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:27,796 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:27,796 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:24:27,813 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:27,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:27,815 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:35,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:35,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:35,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:37,828 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.33 % 2021-12-15 23:24:37,828 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.33 2021-12-15 23:24:37,845 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:37,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:37,848 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:46,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.55 % 2021-12-15 23:24:46,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:46,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:47,861 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.64 % 2021-12-15 23:24:47,861 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 17.64 2021-12-15 23:24:47,875 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:47,878 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:47,878 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:56,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.04 % 2021-12-15 23:24:56,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:24:56,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:24:57,891 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.04 % 2021-12-15 23:24:57,891 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.04 2021-12-15 23:24:57,910 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:24:57,913 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:24:57,913 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:06,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:06,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:06,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:07,926 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:07,926 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 23:25:07,944 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:07,947 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:07,947 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:16,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:16,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:16,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:17,957 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:17,959 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 23:25:17,973 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:17,976 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:17,976 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:26,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:26,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:26,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:27,989 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:27,989 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 23:25:28,006 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:28,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:28,009 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:36,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:36,117 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:36,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:38,023 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.17 % 2021-12-15 23:25:38,023 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.17 2021-12-15 23:25:38,039 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:38,042 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:38,043 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:46,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:25:46,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:46,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:48,055 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:25:48,055 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.39 2021-12-15 23:25:48,072 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:48,075 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:48,075 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:56,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:25:56,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:25:56,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:25:58,086 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:25:58,086 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.39 2021-12-15 23:25:58,104 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:25:58,107 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:25:58,107 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:06,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:26:06,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:06,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:08,119 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.39 % 2021-12-15 23:26:08,119 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.39 2021-12-15 23:26:08,138 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:08,141 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:08,141 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:16,193 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.48 % 2021-12-15 23:26:16,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:16,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:18,153 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.48 % 2021-12-15 23:26:18,153 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.48 2021-12-15 23:26:18,170 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:18,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:18,171 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:26,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.48 % 2021-12-15 23:26:26,239 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:26,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:28,184 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.48 % 2021-12-15 23:26:28,184 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.48 2021-12-15 23:26:28,200 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:28,203 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:28,203 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:36,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.74 % 2021-12-15 23:26:36,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:36,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:38,217 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.74 % 2021-12-15 23:26:38,219 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.74 2021-12-15 23:26:38,244 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:38,252 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:38,252 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:46,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.88 % 2021-12-15 23:26:46,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:46,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:48,266 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.88 % 2021-12-15 23:26:48,266 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 18.88 2021-12-15 23:26:48,282 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:48,285 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:48,285 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:56,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-15 23:26:56,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:26:56,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:26:58,296 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-15 23:26:58,296 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.1 2021-12-15 23:26:58,312 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:26:58,315 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:26:58,315 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:06,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:06,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:06,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:08,326 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:08,328 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.22 2021-12-15 23:27:08,345 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:08,346 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:08,348 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:16,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:16,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:16,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:18,358 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:18,358 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.22 2021-12-15 23:27:18,375 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:18,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:18,378 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:26,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:26,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:26,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:28,394 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-15 23:27:28,394 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.22 2021-12-15 23:27:28,414 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:28,417 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:28,417 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:36,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:36,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:36,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:38,428 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:38,428 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 23:27:38,444 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:38,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:38,447 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:46,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:46,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:46,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:48,457 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:48,457 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 23:27:48,474 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:48,476 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:48,476 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:56,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:56,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:27:56,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:27:58,490 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:27:58,490 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 23:27:58,506 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:27:58,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:27:58,509 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:06,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:28:06,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:28:06,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:08,519 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-15 23:28:08,519 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-15 23:28:08,536 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:28:08,536 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:28:08,538 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:16,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.67 % 2021-12-15 23:28:16,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:28:16,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:18,549 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.67 % 2021-12-15 23:28:18,551 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 19.67 2021-12-15 23:28:18,565 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:28:18,566 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:28:18,566 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:26,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.03 % 2021-12-15 23:28:26,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:28:26,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:28,578 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.03 % 2021-12-15 23:28:28,579 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 20.03 2021-12-15 23:28:28,595 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:28:28,596 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:28:28,598 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:36,492 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.03% in 2896.27800012 seconds 2021-12-15 23:28:36,503 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-15 23:28:36,503 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-15 23:28:36,506 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 23:28:36,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:28:36,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 23:28:36,957 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:28:36,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 23:28:36,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 23:28:36,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 23:28:37,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 23:28:37,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 23:28:37,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 23:28:37,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 23:28:37,059 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-15 23:28:37,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 23:28:37,118 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['203350 memcached\n'], []) 2021-12-15 23:28:37,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-15 23:28:37,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-15 23:28:37,206 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['203350 memcached\n'], []) 2021-12-15 23:28:37,206 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 23:28:37,209 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-15 23:28:37,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:28:37,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-15 23:28:37,677 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:28:37,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-15 23:28:37,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-15 23:28:37,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-15 23:28:37,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-15 23:28:37,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-15 23:28:37,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-15 23:28:37,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-15 23:28:37,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:28:37,822 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-15 23:28:37,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:28:37,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:37,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:37,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:37,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:37,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:28:37,967 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-15 23:28:37,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:28:37,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:37,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:28:38,007 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-15 23:28:38,007 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-15 23:28:38,010 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 23:28:38,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:28:38,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 23:28:38,470 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:28:38,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 23:28:38,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 23:28:38,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 23:28:38,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 23:28:38,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 23:28:38,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 23:28:38,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 23:28:38,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:28:38,611 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.03 % 2021-12-15 23:28:38,611 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 20.03 2021-12-15 23:28:38,631 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 23:28:38,634 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:28:38,635 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:28:38,635 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:38,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:28:38,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:38,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:38,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:38,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:38,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:38,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:38,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:38,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:38,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:39,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:39,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:39,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:39,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:39,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:39,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:28:39,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:28:39,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:28:39,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:28:39,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:28:39,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:28:39,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:28:39,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:28:39,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:39,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:39,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:28:39,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:28:39,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:28:40,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:28:40,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:28:40,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:28:40,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:28:40,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:28:40,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:28:40,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:28:40,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:28:40,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:28:40,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:40,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 23:28:40,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 23:28:40,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:28:40,898 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 23:28:40,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:28:40,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:40,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:28:40,947 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 23:28:40,948 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 23:28:40,950 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 23:28:41,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:28:41,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 23:28:41,437 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:28:41,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 23:28:41,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 23:28:41,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 23:28:41,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 23:28:41,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 23:28:41,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 23:28:41,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 23:28:41,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:28:41,553 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 23:28:41,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:28:41,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:41,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:41,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:41,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:41,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:41,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:41,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:42,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:42,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:42,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:42,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:42,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:42,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:42,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:42,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:28:42,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:28:42,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:28:42,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:28:42,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:42,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:42,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:28:42,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:28:43,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:28:43,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:28:43,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:28:43,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:28:43,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:28:43,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:28:43,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:28:43,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:28:43,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:28:43,752 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:28:43,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:28:43,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:28:43,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:43,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:43,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:28:43,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:28:44,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:28:44,023 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 23:28:44,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:28:44,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:44,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:28:44,065 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 23:28:44,065 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 23:28:44,069 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 23:28:44,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:28:44,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 23:28:44,602 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:28:44,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 23:28:44,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 23:28:44,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 23:28:44,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 23:28:44,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 23:28:44,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 23:28:44,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 23:28:44,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:28:44,752 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 23:28:44,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:28:44,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:44,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:44,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:44,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:28:44,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:44,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:44,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:44,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:28:45,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:45,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:45,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:45,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:28:45,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:45,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:45,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:45,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:28:45,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:45,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:28:45,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:45,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:28:45,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:28:45,375 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 23:28:45,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:28:45,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:28:45,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:28:45,424 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 23:28:45,424 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 23:28:45,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-15 23:28:45,430 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-15 23:28:48,651 | test | DEBUG | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.03 % 2021-12-15 23:28:48,651 | test | INFO | pool-3-thread-30 | [task:check:385] Rebalance - status: running, progress: 20.03 2021-12-15 23:28:48,676 | infra | DEBUG | pool-3-thread-30 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:28:48,677 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:28:48,677 | infra | DEBUG | pool-3-thread-30 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:28:58,687 | test | ERROR | pool-3-thread-30 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9ed58a2b24f35e1dd40286e75f56c876', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4c62aa5224bffcceb451a33a15395d17', u'status': u'notRunning'} - rebalance failed 2021-12-15 23:28:58,713 | test | INFO | pool-3-thread-30 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-15 23:28:58,713 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730609L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.609Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 22:32:21.577992 203578 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-15 23:28:58,714 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730593L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.593Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6165.169>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,86,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.65:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.5618.169>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}}.\nRebalance Operation Id = a06304b5801ad90f607f24ba9292d3d5'} 2021-12-15 23:28:58,714 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730571L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.571Z', u'text': u'Worker <0.5277.169> (for action {move,{86,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.6165.169>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 86,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.65:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.5618.169>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730559L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.559Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639636752409L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:39:12.409Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.67', u'tstamp': 1639636739948L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:38:59.948Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.67\' in 0 seconds.'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1639636739925L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:38:59.925Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.68\' in 0 seconds.'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639636739536L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:38:59.536Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.110.68', u'tstamp': 1639636738954L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:38:58.954Z', u'text': u'Deleting old data files of bucket "GleamBookUsers0"'} 2021-12-15 23:28:58,716 | test | ERROR | pool-3-thread-30 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.67', u'tstamp': 1639636738644L, u'shortText': u'message', u'serverTime': u'2021-12-15T22:38:58.644Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2021-12-15 23:28:58,717 | test | ERROR | pool-3-thread-30 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'9ed58a2b24f35e1dd40286e75f56c876', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=4c62aa5224bffcceb451a33a15395d17', u'status': u'notRunning'} - rebalance failed 2021-12-15 23:29:45,444 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-15 23:29:45,446 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 23:29:45,457 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-15 23:29:45,469 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-15 23:29:45,469 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 23:29:45,477 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:45,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:45,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:47,497 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:47,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:47,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:49,515 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:49,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:49,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:51,536 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:51,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:51,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:53,552 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:53,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:53,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:55,571 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:55,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:55,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:57,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:57,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:57,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:29:59,609 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:29:59,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:29:59,617 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:01,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:01,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:01,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:03,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:03,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:03,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:05,667 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:05,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:05,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:07,690 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:07,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:07,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:09,710 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:09,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:09,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:11,732 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:11,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:11,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:13,752 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:13,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:13,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:15,772 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:15,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:15,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:17,790 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:17,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:17,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:19,812 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:19,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:19,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:21,829 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:21,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:21,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:23,848 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:23,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:23,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:25,867 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:25,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:25,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:27,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:27,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:27,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:29,901 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:29,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:29,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:31,918 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:31,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:31,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:33,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:33,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:33,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:35,954 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:35,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:35,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:37,971 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:37,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:37,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:39,990 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:39,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:39,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:42,007 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:42,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:42,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:44,026 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:44,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:44,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:46,045 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:46,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:46,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:48,065 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:48,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:48,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:50,084 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:50,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:50,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:52,104 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:52,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:52,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:54,121 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:54,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:54,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:56,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:56,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:56,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:30:58,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:30:58,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:30:58,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:00,177 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:00,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:00,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:02,196 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:02,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:02,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:04,217 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:04,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:04,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:06,236 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:06,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:06,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:08,255 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:08,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:08,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:10,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:10,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:10,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:12,298 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:12,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:12,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:14,322 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:14,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:14,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:16,344 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:16,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:16,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:18,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:18,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:18,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:20,384 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:20,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:20,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:22,404 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:22,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:22,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:24,421 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:24,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:24,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:26,438 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:26,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:26,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:28,456 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:28,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:28,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:30,474 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:30,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:30,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:32,493 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:32,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:32,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:34,516 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:34,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:34,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:36,533 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:36,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:36,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:38,555 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:38,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:38,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:43,092 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:43,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:43,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:45,112 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:45,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:45,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:47,131 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:47,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:47,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:49,151 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:49,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:49,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:51,168 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:51,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:51,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:53,186 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:53,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:53,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:55,206 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:55,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:55,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:57,226 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:57,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:57,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:31:59,246 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:31:59,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:31:59,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:01,266 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:01,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:01,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:03,289 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:03,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:03,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:05,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:05,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:05,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:07,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:07,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:07,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:09,349 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:09,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:09,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:11,369 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:11,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:11,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:13,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:13,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:13,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:15,411 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:15,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:15,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:17,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:17,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:17,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:19,454 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:19,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:19,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:21,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:21,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:21,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:23,500 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:23,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:23,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:25,519 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:25,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:25,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:27,543 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:27,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:27,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:29,566 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:29,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:29,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:31,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:31,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:31,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:33,607 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:33,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:33,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:35,627 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:35,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:35,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:37,645 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:37,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:37,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:39,665 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:39,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:39,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:41,686 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-15 23:32:41,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-15 23:32:41,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-15 23:32:43,703 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-15 23:32:43,713 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-15 23:32:43,713 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 23:32:43,723 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-15 23:32:43,733 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-15 23:32:43,733 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-15 23:32:43,746 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-15 23:32:43,746 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-15 23:32:43,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:32:44,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-15 23:32:44,243 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:32:44,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-15 23:32:44,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-15 23:32:44,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-15 23:32:44,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-15 23:32:44,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-15 23:32:44,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-15 23:32:44,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-15 23:32:44,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:32:44,391 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-15 23:32:44,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:32:44,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:44,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:44,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:44,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:44,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:32:44,493 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-15 23:32:44,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:32:44,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:44,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:32:44,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-15 23:32:44,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-15 23:32:44,532 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-15 23:32:44,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:32:44,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-15 23:32:44,953 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:32:44,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-15 23:32:44,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-15 23:32:44,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-15 23:32:45,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-15 23:32:45,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-15 23:32:45,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-15 23:32:45,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-15 23:32:45,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:32:45,230 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-15 23:32:45,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:32:45,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:45,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:45,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:45,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:45,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:45,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:45,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:45,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:45,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:45,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:45,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:45,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:45,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:32:46,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:32:46,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:32:46,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:32:46,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:32:46,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:32:46,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:32:46,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:32:46,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:32:46,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:32:46,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:32:46,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:32:46,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:32:46,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:32:46,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:32:46,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:32:46,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:32:46,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:32:46,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:46,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:32:46,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:32:46,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:46,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:47,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 23:32:47,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-15 23:32:47,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:47,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:47,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-15 23:32:47,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-15 23:32:47,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:32:47,140 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-15 23:32:47,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:32:47,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:47,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:32:47,203 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-15 23:32:47,203 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-15 23:32:47,207 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-15 23:32:47,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:32:47,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-15 23:32:47,575 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:32:47,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-15 23:32:47,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-15 23:32:47,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-15 23:32:47,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-15 23:32:47,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-15 23:32:47,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-15 23:32:47,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-15 23:32:47,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:32:47,690 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-15 23:32:47,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:32:47,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:47,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:47,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:47,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:47,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:47,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:47,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:47,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:47,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:48,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:48,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:48,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:48,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:48,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:48,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:32:48,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-15 23:32:48,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:32:48,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-15 23:32:48,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:32:48,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-15 23:32:48,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:32:48,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-15 23:32:48,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:32:48,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-15 23:32:48,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:32:48,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-15 23:32:48,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:32:48,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-15 23:32:48,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:48,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:32:48,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-15 23:32:48,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:48,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:49,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:32:49,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-15 23:32:49,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:49,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:49,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:32:49,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-15 23:32:49,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:32:49,124 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-15 23:32:49,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:32:49,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:49,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:32:49,153 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-15 23:32:49,153 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-15 23:32:49,155 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-15 23:32:49,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-15 23:32:49,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-15 23:32:49,565 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-15 23:32:49,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-15 23:32:49,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-15 23:32:49,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-15 23:32:49,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-15 23:32:49,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-15 23:32:49,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-15 23:32:49,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-15 23:32:49,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-15 23:32:49,676 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-15 23:32:49,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-15 23:32:49,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:49,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:49,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:49,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-15 23:32:49,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:49,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:49,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:49,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-15 23:32:49,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:49,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:50,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:50,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-15 23:32:50,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:50,076 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:50,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:50,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-15 23:32:50,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:50,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-15 23:32:50,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:50,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-15 23:32:50,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-15 23:32:50,240 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-15 23:32:50,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-15 23:32:50,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-15 23:32:50,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-15 23:32:50,282 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-15 23:32:50,282 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-15 23:32:50,282 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.68,172.23.110.67]_OUT=[172.23.110.66]_1639636730.15 2021-12-15 23:32:50,283 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-15 23:32:50,329 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639639970.29 2021-12-15 23:32:50,329 | infra | INFO | pool-3-thread-1 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639639970.29' started 2021-12-15 23:32:50,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-15 23:32:50,371 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-15 23:32:50,378 | test | DEBUG | pool-3-thread-1 | [task:start_rebalance:356] Removing node 172.23.110.66:8091 from cluster 2021-12-15 23:32:50,378 | test | DEBUG | pool-3-thread-1 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.66', 'user': 'Administrator'} 2021-12-15 23:32:50,434 | test | DEBUG | pool-3-thread-1 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-15 23:32:50,440 | test | INFO | pool-3-thread-1 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 0.156957524238 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 3.27636318124 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 1.45016283018 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 8.09469369545 | --- OUT ---> | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 6.72983140588 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-15 23:32:50,450 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:32:50,450 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:32:50,470 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:32:50,471 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:32:50,473 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:00,483 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:33:00,483 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:33:00,509 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:00,513 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:00,513 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:10,526 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:33:10,526 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:33:10,546 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:10,551 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:10,551 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:20,572 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:33:20,572 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:33:20,591 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:20,592 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:20,592 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:30,604 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:33:30,605 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:33:30,622 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:30,625 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:30,625 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:40,635 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-15 23:33:40,637 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-15 23:33:40,654 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:40,657 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:40,657 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:33:50,670 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.13 % 2021-12-15 23:33:50,670 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.13 2021-12-15 23:33:50,687 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:33:50,690 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:33:50,690 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:00,707 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.45 % 2021-12-15 23:34:00,707 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.45 2021-12-15 23:34:00,726 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:00,729 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:00,729 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:10,746 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.45 % 2021-12-15 23:34:10,746 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.45 2021-12-15 23:34:10,766 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:10,767 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:10,767 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:20,779 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.45 % 2021-12-15 23:34:20,779 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.45 2021-12-15 23:34:20,796 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:20,799 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:20,799 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:30,812 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.63 % 2021-12-15 23:34:30,812 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.63 2021-12-15 23:34:30,832 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:30,834 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:30,834 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:40,845 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:34:40,845 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.92 2021-12-15 23:34:40,865 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:40,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:40,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:50,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:34:50,384 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.92 2021-12-15 23:34:50,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:34:50,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:34:50,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:34:50,878 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:34:50,878 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.92 2021-12-15 23:34:50,903 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:34:50,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:34:50,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:00,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:35:00,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:00,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:00,917 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.92 % 2021-12-15 23:35:00,917 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 0.92 2021-12-15 23:35:00,934 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:00,936 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:00,936 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:10,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.24 % 2021-12-15 23:35:10,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:10,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:10,947 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.24 % 2021-12-15 23:35:10,947 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.24 2021-12-15 23:35:10,966 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:10,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:10,969 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:20,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 23:35:20,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:20,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:20,980 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-15 23:35:20,980 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.37 2021-12-15 23:35:21,000 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:21,003 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:21,003 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:30,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:30,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:30,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:31,016 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:31,016 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.55 2021-12-15 23:35:31,036 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:31,038 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:31,038 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:40,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:40,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:40,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:41,049 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:41,049 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.55 2021-12-15 23:35:41,065 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:41,066 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:41,068 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:50,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:50,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:35:50,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:35:51,079 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:35:51,079 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.55 2021-12-15 23:35:51,101 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:35:51,101 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:35:51,102 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:00,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:36:00,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:00,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:01,114 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:36:01,114 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.55 2021-12-15 23:36:01,130 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:01,131 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:01,131 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:10,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:36:10,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:10,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:11,144 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.55 % 2021-12-15 23:36:11,145 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.55 2021-12-15 23:36:11,173 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:11,176 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:11,176 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:20,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.83 % 2021-12-15 23:36:20,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:20,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:21,187 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.83 % 2021-12-15 23:36:21,187 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 1.83 2021-12-15 23:36:21,207 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:21,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:21,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:30,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 23:36:30,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:30,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:31,220 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.11 % 2021-12-15 23:36:31,221 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.11 2021-12-15 23:36:31,239 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:31,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:31,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:40,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.47 % 2021-12-15 23:36:40,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:40,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:41,253 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.47 % 2021-12-15 23:36:41,253 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.47 2021-12-15 23:36:41,272 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:41,273 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:41,273 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:50,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.47 % 2021-12-15 23:36:50,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:36:50,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:36:51,285 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.47 % 2021-12-15 23:36:51,286 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.47 2021-12-15 23:36:51,303 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:36:51,306 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:36:51,306 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:00,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.65 % 2021-12-15 23:37:00,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:00,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:01,318 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.65 % 2021-12-15 23:37:01,319 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.65 2021-12-15 23:37:01,335 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:01,336 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:01,336 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:10,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.65 % 2021-12-15 23:37:10,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:10,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:11,352 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.65 % 2021-12-15 23:37:11,352 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.65 2021-12-15 23:37:11,372 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:11,377 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:11,378 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:20,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:20,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:20,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:21,391 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:21,391 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.79 2021-12-15 23:37:21,410 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:21,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:21,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:30,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:30,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:30,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:31,427 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:31,427 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.79 2021-12-15 23:37:31,450 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:31,453 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:31,453 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:40,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:40,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:40,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:41,466 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-15 23:37:41,466 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 2.79 2021-12-15 23:37:41,482 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:41,484 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:41,486 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:50,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.10 % 2021-12-15 23:37:50,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:37:50,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:37:51,497 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.10 % 2021-12-15 23:37:51,497 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.1 2021-12-15 23:37:51,515 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:37:51,516 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:37:51,517 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:00,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.25 % 2021-12-15 23:38:00,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:00,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:01,529 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.25 % 2021-12-15 23:38:01,530 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.25 2021-12-15 23:38:01,549 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:01,552 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:01,552 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:10,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.39 % 2021-12-15 23:38:10,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:10,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:11,565 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.39 % 2021-12-15 23:38:11,565 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.39 2021-12-15 23:38:11,584 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:11,588 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:11,588 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:20,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 23:38:20,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:20,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:21,605 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 23:38:21,605 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 23:38:21,638 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:21,641 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:21,641 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:30,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 23:38:30,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:30,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:31,654 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-15 23:38:31,654 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-15 23:38:31,671 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:31,674 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:31,674 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:40,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.89 % 2021-12-15 23:38:40,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:40,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:41,686 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.89 % 2021-12-15 23:38:41,687 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.89 2021-12-15 23:38:41,704 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:41,707 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:41,707 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:50,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.89 % 2021-12-15 23:38:50,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:38:50,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:38:51,719 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.89 % 2021-12-15 23:38:51,719 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 3.89 2021-12-15 23:38:51,742 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:38:51,743 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:38:51,743 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:00,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.07 % 2021-12-15 23:39:00,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:00,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:01,755 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.07 % 2021-12-15 23:39:01,756 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.07 2021-12-15 23:39:01,773 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:01,775 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:01,775 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:10,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 23:39:10,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:10,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:11,786 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 23:39:11,786 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.2 2021-12-15 23:39:11,803 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:11,805 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:11,805 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:20,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 23:39:20,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:20,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:21,816 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.20 % 2021-12-15 23:39:21,818 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.2 2021-12-15 23:39:21,839 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:21,842 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:21,842 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:30,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.34 % 2021-12-15 23:39:30,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:30,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:31,854 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.34 % 2021-12-15 23:39:31,855 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.34 2021-12-15 23:39:31,871 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:31,872 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:31,872 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:40,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.34 % 2021-12-15 23:39:40,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:40,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:41,884 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.34 % 2021-12-15 23:39:41,884 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.34 2021-12-15 23:39:41,900 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:41,901 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:41,901 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:50,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.48 % 2021-12-15 23:39:50,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:39:50,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:39:51,915 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.48 % 2021-12-15 23:39:51,917 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.48 2021-12-15 23:39:51,933 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:39:51,934 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:39:51,934 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:01,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.63 % 2021-12-15 23:40:01,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:01,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:01,947 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.63 % 2021-12-15 23:40:01,947 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.63 2021-12-15 23:40:01,966 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:01,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:01,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:11,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.99 % 2021-12-15 23:40:11,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:11,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:11,980 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.99 % 2021-12-15 23:40:11,980 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.99 2021-12-15 23:40:11,996 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:11,997 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:11,997 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:21,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.99 % 2021-12-15 23:40:21,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:21,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:22,013 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.99 % 2021-12-15 23:40:22,015 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 4.99 2021-12-15 23:40:22,038 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:22,040 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:22,040 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:31,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.99 % 2021-12-15 23:40:31,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:31,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:32,053 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.12 % 2021-12-15 23:40:32,053 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.12 2021-12-15 23:40:32,076 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:32,081 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:32,081 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:41,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:40:41,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:41,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:42,096 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:40:42,096 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.44 2021-12-15 23:40:42,121 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:42,122 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:42,122 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:51,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:40:51,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:40:51,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:40:52,135 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:40:52,135 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.44 2021-12-15 23:40:52,155 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:40:52,157 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:40:52,157 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:01,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:41:01,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:01,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:02,170 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.44 % 2021-12-15 23:41:02,170 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.44 2021-12-15 23:41:02,187 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:02,190 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:02,190 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:11,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.62 % 2021-12-15 23:41:11,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:11,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:12,203 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.62 % 2021-12-15 23:41:12,203 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.62 2021-12-15 23:41:12,220 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:12,221 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:12,223 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:21,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.62 % 2021-12-15 23:41:21,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:21,165 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:22,234 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.62 % 2021-12-15 23:41:22,234 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.62 2021-12-15 23:41:22,250 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:22,253 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:22,253 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:31,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.77 % 2021-12-15 23:41:31,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:31,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:32,265 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.77 % 2021-12-15 23:41:32,265 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 5.77 2021-12-15 23:41:32,280 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:32,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:32,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:41,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.04 % 2021-12-15 23:41:41,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:41,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:42,305 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.04 % 2021-12-15 23:41:42,305 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.04 2021-12-15 23:41:42,323 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:42,326 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:42,326 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:51,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.18 % 2021-12-15 23:41:51,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:41:51,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:41:52,338 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.18 % 2021-12-15 23:41:52,338 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.18 2021-12-15 23:41:52,355 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:41:52,357 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:41:52,357 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:01,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.36 % 2021-12-15 23:42:01,247 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:01,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:02,368 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.36 % 2021-12-15 23:42:02,368 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.36 2021-12-15 23:42:02,385 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:02,388 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:02,388 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:11,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.54 % 2021-12-15 23:42:11,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:11,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:12,400 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.54 % 2021-12-15 23:42:12,400 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.54 2021-12-15 23:42:12,417 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:12,420 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:12,421 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:21,280 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.54 % 2021-12-15 23:42:21,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:21,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:22,431 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.54 % 2021-12-15 23:42:22,433 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.54 2021-12-15 23:42:22,448 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:22,450 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:22,451 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:31,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.72 % 2021-12-15 23:42:31,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:31,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:32,463 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.72 % 2021-12-15 23:42:32,463 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.72 2021-12-15 23:42:32,480 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:32,483 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:32,483 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:41,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:42:41,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:41,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:42,494 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:42:42,494 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.86 2021-12-15 23:42:42,510 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:42,515 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:42,515 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:51,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:42:51,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:42:51,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:42:52,526 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:42:52,526 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.86 2021-12-15 23:42:52,542 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:42:52,546 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:42:52,546 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:01,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:43:01,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:01,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:02,558 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.86 % 2021-12-15 23:43:02,559 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.86 2021-12-15 23:43:02,575 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:02,578 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:02,578 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:11,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.99 % 2021-12-15 23:43:11,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:11,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:12,592 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.99 % 2021-12-15 23:43:12,592 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 6.99 2021-12-15 23:43:12,611 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:12,615 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:12,615 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:21,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.28 % 2021-12-15 23:43:21,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:21,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:22,628 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.46 % 2021-12-15 23:43:22,628 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.46 2021-12-15 23:43:22,647 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:22,651 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:22,651 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:31,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.46 % 2021-12-15 23:43:31,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:31,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:32,665 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.46 % 2021-12-15 23:43:32,667 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.46 2021-12-15 23:43:32,686 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:32,690 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:32,690 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:41,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.46 % 2021-12-15 23:43:41,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:41,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:42,704 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.46 % 2021-12-15 23:43:42,704 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.46 2021-12-15 23:43:42,726 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:42,730 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:42,730 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:51,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:43:51,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:43:51,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:43:52,743 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.59 % 2021-12-15 23:43:52,743 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.59 2021-12-15 23:43:52,763 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:43:52,767 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:43:52,767 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:01,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.78 % 2021-12-15 23:44:01,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:01,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:02,780 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.78 % 2021-12-15 23:44:02,780 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.78 2021-12-15 23:44:02,809 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:02,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:02,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:11,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.78 % 2021-12-15 23:44:11,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:11,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:12,825 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.78 % 2021-12-15 23:44:12,825 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.78 2021-12-15 23:44:12,842 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:12,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:12,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:21,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.78 % 2021-12-15 23:44:21,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:21,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:22,857 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.96 % 2021-12-15 23:44:22,857 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 7.96 2021-12-15 23:44:22,872 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:22,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:22,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:31,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.09 % 2021-12-15 23:44:31,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:31,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:32,887 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.28 % 2021-12-15 23:44:32,888 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.28 2021-12-15 23:44:32,904 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:32,905 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:32,905 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:41,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.28 % 2021-12-15 23:44:41,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:41,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:42,917 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.28 % 2021-12-15 23:44:42,917 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.28 2021-12-15 23:44:42,934 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:42,936 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:42,936 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:51,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.28 % 2021-12-15 23:44:51,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:44:51,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:44:52,948 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.41 % 2021-12-15 23:44:52,948 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.41 2021-12-15 23:44:52,973 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:44:52,974 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:44:52,974 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:01,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.41 % 2021-12-15 23:45:01,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:01,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:02,984 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.41 % 2021-12-15 23:45:02,984 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.41 2021-12-15 23:45:03,000 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:03,002 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:03,002 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:11,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:11,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:11,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:13,012 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:13,013 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.7 2021-12-15 23:45:13,028 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:13,029 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:13,029 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:21,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:21,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:21,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:23,043 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:23,043 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.7 2021-12-15 23:45:23,063 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:23,065 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:23,065 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:31,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:31,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:31,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:33,078 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.70 % 2021-12-15 23:45:33,079 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.7 2021-12-15 23:45:33,098 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:33,101 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:33,101 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:41,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.88 % 2021-12-15 23:45:41,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:41,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:43,112 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.88 % 2021-12-15 23:45:43,114 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 8.88 2021-12-15 23:45:43,132 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:43,134 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:43,134 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:51,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.20 % 2021-12-15 23:45:51,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:45:51,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:45:53,150 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.20 % 2021-12-15 23:45:53,151 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.2 2021-12-15 23:45:53,167 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:45:53,171 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:45:53,171 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:01,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.20 % 2021-12-15 23:46:01,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:01,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:03,181 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.20 % 2021-12-15 23:46:03,183 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.2 2021-12-15 23:46:03,200 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:03,203 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:03,203 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:11,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.20 % 2021-12-15 23:46:11,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:11,770 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:13,217 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.51 % 2021-12-15 23:46:13,217 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.51 2021-12-15 23:46:13,236 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:13,239 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:13,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:21,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.51 % 2021-12-15 23:46:21,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:21,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:23,252 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.51 % 2021-12-15 23:46:23,253 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.51 2021-12-15 23:46:23,270 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:23,273 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:23,273 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:31,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.51 % 2021-12-15 23:46:31,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:31,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:33,285 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.51 % 2021-12-15 23:46:33,285 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.51 2021-12-15 23:46:33,301 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:33,302 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:33,302 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:41,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.69 % 2021-12-15 23:46:41,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:41,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:43,312 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.69 % 2021-12-15 23:46:43,313 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.69 2021-12-15 23:46:43,329 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:43,332 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:43,332 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:51,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.84 % 2021-12-15 23:46:51,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:46:51,857 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:46:53,344 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.84 % 2021-12-15 23:46:53,345 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.84 2021-12-15 23:46:53,364 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:46:53,367 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:46:53,367 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:01,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.98 % 2021-12-15 23:47:01,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:01,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:03,378 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.98 % 2021-12-15 23:47:03,378 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 9.98 2021-12-15 23:47:03,394 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:03,395 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:03,395 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:11,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.11 % 2021-12-15 23:47:11,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:11,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:13,407 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-15 23:47:13,407 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-15 23:47:13,424 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:13,426 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:13,426 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:21,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-15 23:47:21,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:21,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:23,438 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-15 23:47:23,440 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-15 23:47:23,457 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:23,459 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:23,459 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:31,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-15 23:47:31,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:31,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:33,470 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-15 23:47:33,470 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-15 23:47:33,487 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:33,490 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:33,490 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:41,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.43 % 2021-12-15 23:47:41,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:41,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:43,503 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.61 % 2021-12-15 23:47:43,505 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.61 2021-12-15 23:47:43,520 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:43,523 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:43,523 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:51,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.61 % 2021-12-15 23:47:51,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:47:51,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:47:53,536 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.61 % 2021-12-15 23:47:53,536 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.61 2021-12-15 23:47:53,555 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:47:53,558 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:47:53,558 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:01,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.61 % 2021-12-15 23:48:01,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:01,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:03,571 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.61 % 2021-12-15 23:48:03,572 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.61 2021-12-15 23:48:03,591 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:03,595 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:03,595 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:11,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.80 % 2021-12-15 23:48:11,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:11,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:13,608 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.80 % 2021-12-15 23:48:13,608 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.8 2021-12-15 23:48:13,625 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:13,628 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:13,628 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:21,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.93 % 2021-12-15 23:48:22,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:22,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:23,641 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.93 % 2021-12-15 23:48:23,641 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 10.93 2021-12-15 23:48:23,660 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:23,663 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:23,664 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:32,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.06 % 2021-12-15 23:48:32,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:32,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:33,676 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.21 % 2021-12-15 23:48:33,677 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.21 2021-12-15 23:48:33,694 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:33,697 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:33,697 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:42,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:48:42,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:42,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:43,709 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:48:43,709 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.53 2021-12-15 23:48:43,724 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:43,727 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:43,727 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:52,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:48:52,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:48:52,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:48:53,740 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:48:53,740 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.53 2021-12-15 23:48:53,756 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:48:53,759 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:48:53,760 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:02,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:49:02,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:02,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:03,772 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:49:03,772 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.53 2021-12-15 23:49:03,805 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:03,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:03,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:12,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:49:12,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:12,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:13,823 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.53 % 2021-12-15 23:49:13,823 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.53 2021-12-15 23:49:13,842 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:13,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:13,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:22,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.67 % 2021-12-15 23:49:22,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:22,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:23,857 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.67 % 2021-12-15 23:49:23,857 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.67 2021-12-15 23:49:23,875 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:23,877 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:23,878 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:32,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.67 % 2021-12-15 23:49:32,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:32,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:33,891 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.67 % 2021-12-15 23:49:33,891 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 11.67 2021-12-15 23:49:33,907 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:33,908 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:33,908 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:42,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.03 % 2021-12-15 23:49:42,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:42,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:43,920 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.03 % 2021-12-15 23:49:43,920 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.03 2021-12-15 23:49:43,937 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:43,938 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:43,938 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:52,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.16 % 2021-12-15 23:49:52,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:49:52,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:49:53,950 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.16 % 2021-12-15 23:49:53,950 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.16 2021-12-15 23:49:53,967 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:49:53,969 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:49:53,969 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:02,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.16 % 2021-12-15 23:50:02,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:02,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:03,983 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.16 % 2021-12-15 23:50:03,983 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.16 2021-12-15 23:50:04,002 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:04,005 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:04,006 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:12,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.35 % 2021-12-15 23:50:12,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:12,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:14,017 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.35 % 2021-12-15 23:50:14,019 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.35 2021-12-15 23:50:14,039 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:14,040 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:14,040 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:22,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.62 % 2021-12-15 23:50:22,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:22,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:24,052 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.62 % 2021-12-15 23:50:24,052 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.62 2021-12-15 23:50:24,069 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:24,071 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:24,071 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:32,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:32,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:32,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:34,082 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:34,082 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.95 2021-12-15 23:50:34,098 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:34,098 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:34,098 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:42,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:42,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:42,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:44,111 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:44,112 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.95 2021-12-15 23:50:44,131 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:44,134 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:44,134 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:52,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:52,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:50:52,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:50:54,144 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:50:54,145 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.95 2021-12-15 23:50:54,164 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:50:54,165 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:50:54,165 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:02,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:51:02,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:02,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:04,180 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:51:04,180 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.95 2021-12-15 23:51:04,197 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:04,201 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:04,201 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:12,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:51:12,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:12,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:14,213 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.95 % 2021-12-15 23:51:14,213 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 12.95 2021-12-15 23:51:14,232 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:14,234 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:14,234 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:22,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.08 % 2021-12-15 23:51:22,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:22,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:24,247 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.08 % 2021-12-15 23:51:24,247 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 13.08 2021-12-15 23:51:24,265 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:24,266 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:24,266 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:32,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.40 % 2021-12-15 23:51:32,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:32,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:34,280 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.40 % 2021-12-15 23:51:34,282 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 13.4 2021-12-15 23:51:34,299 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:34,303 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:34,305 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:42,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-15 23:51:42,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:42,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:44,315 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-15 23:51:44,315 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 13.58 2021-12-15 23:51:44,332 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:44,335 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:44,335 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:52,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-15 23:51:52,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:51:52,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:51:54,346 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-15 23:51:54,346 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 13.58 2021-12-15 23:51:54,368 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:51:54,384 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:51:54,385 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:02,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.77 % 2021-12-15 23:52:02,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:02,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:04,404 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.77 % 2021-12-15 23:52:04,404 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 13.77 2021-12-15 23:52:04,428 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:04,433 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:04,433 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:12,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-15 23:52:12,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:12,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:14,444 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-15 23:52:14,444 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.05 2021-12-15 23:52:14,460 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:14,463 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:14,463 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:22,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-15 23:52:22,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:22,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:24,476 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-15 23:52:24,476 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.05 2021-12-15 23:52:24,493 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:24,493 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:24,494 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:32,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-15 23:52:32,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:32,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:34,506 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.19 % 2021-12-15 23:52:34,506 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.19 2021-12-15 23:52:34,522 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:34,523 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:34,525 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:42,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:52:42,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:42,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:44,536 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:52:44,536 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.5 2021-12-15 23:52:44,555 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:44,558 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:44,558 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:52,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:52:52,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:52:52,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:52:54,569 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:52:54,569 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.5 2021-12-15 23:52:54,585 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:52:54,585 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:52:54,586 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:02,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:53:02,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:02,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:04,598 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:53:04,598 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.5 2021-12-15 23:53:04,614 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:04,617 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:04,617 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:12,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:53:12,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:12,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:14,628 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.50 % 2021-12-15 23:53:14,628 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.5 2021-12-15 23:53:14,644 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:14,648 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:14,648 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:22,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.68 % 2021-12-15 23:53:22,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:22,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:24,661 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.68 % 2021-12-15 23:53:24,664 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.68 2021-12-15 23:53:24,684 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:24,686 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:24,686 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:32,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.68 % 2021-12-15 23:53:32,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:32,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:34,697 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.68 % 2021-12-15 23:53:34,697 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 14.68 2021-12-15 23:53:34,713 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:34,714 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:34,716 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:42,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.87 % 2021-12-15 23:53:42,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:42,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:44,726 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.18 % 2021-12-15 23:53:44,726 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.18 2021-12-15 23:53:44,743 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:44,744 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:44,746 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:52,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.46 % 2021-12-15 23:53:52,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:53:52,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:53:54,757 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.46 % 2021-12-15 23:53:54,757 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.46 2021-12-15 23:53:54,773 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:53:54,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:53:54,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:02,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:02,660 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:02,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:04,789 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:04,789 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.6 2021-12-15 23:54:04,805 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:04,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:04,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:12,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:12,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:12,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:14,819 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:14,819 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.6 2021-12-15 23:54:14,836 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:14,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:14,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:22,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:22,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:22,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:24,851 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:24,851 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.6 2021-12-15 23:54:24,869 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:24,871 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:24,871 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:32,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:32,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:32,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:34,887 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.60 % 2021-12-15 23:54:34,887 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.6 2021-12-15 23:54:34,907 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:34,911 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:34,911 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:42,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.79 % 2021-12-15 23:54:42,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:42,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:44,921 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.79 % 2021-12-15 23:54:44,921 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.79 2021-12-15 23:54:44,937 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:44,940 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:44,940 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:52,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.92 % 2021-12-15 23:54:52,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:54:52,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:54:54,950 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.92 % 2021-12-15 23:54:54,950 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 15.92 2021-12-15 23:54:54,967 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:54:54,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:54:54,969 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:02,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.05 % 2021-12-15 23:55:02,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:02,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:04,980 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.05 % 2021-12-15 23:55:04,980 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.05 2021-12-15 23:55:04,997 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:05,000 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:05,000 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:12,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.24 % 2021-12-15 23:55:12,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:12,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:15,012 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.24 % 2021-12-15 23:55:15,013 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.24 2021-12-15 23:55:15,030 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:15,033 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:15,035 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:22,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.24 % 2021-12-15 23:55:22,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:22,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:25,049 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.24 % 2021-12-15 23:55:25,049 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.24 2021-12-15 23:55:25,073 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:25,078 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:25,078 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:32,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.24 % 2021-12-15 23:55:32,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:32,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:35,091 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-15 23:55:35,091 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.38 2021-12-15 23:55:35,109 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:35,111 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:35,111 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:42,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.89 % 2021-12-15 23:55:42,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:42,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:45,127 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.89 % 2021-12-15 23:55:45,128 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.89 2021-12-15 23:55:45,145 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:45,150 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:45,150 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:52,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.89 % 2021-12-15 23:55:52,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:55:52,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:55:55,163 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.89 % 2021-12-15 23:55:55,163 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 16.89 2021-12-15 23:55:55,177 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:55:55,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:55:55,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:02,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.89 % 2021-12-15 23:56:02,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:02,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:05,194 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.02 % 2021-12-15 23:56:05,194 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.02 2021-12-15 23:56:05,210 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:05,211 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:05,211 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:12,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.02 % 2021-12-15 23:56:12,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:12,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:15,224 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.16 % 2021-12-15 23:56:15,224 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.16 2021-12-15 23:56:15,246 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:15,249 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:15,249 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:22,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.16 % 2021-12-15 23:56:22,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:22,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:25,266 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.16 % 2021-12-15 23:56:25,266 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.16 2021-12-15 23:56:25,285 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:25,288 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:25,288 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:32,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:32,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:32,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:35,302 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:35,302 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.34 2021-12-15 23:56:35,321 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:35,322 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:35,322 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:42,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:43,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:43,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:45,338 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:45,338 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.34 2021-12-15 23:56:45,355 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:45,358 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:45,358 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:53,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:53,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:56:53,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:56:55,371 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.34 % 2021-12-15 23:56:55,371 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.34 2021-12-15 23:56:55,388 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:56:55,391 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:56:55,391 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:03,038 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.52 % 2021-12-15 23:57:03,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:03,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:05,405 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.52 % 2021-12-15 23:57:05,405 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.52 2021-12-15 23:57:05,423 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:05,426 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:05,426 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:13,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.65 % 2021-12-15 23:57:13,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:13,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:15,444 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.80 % 2021-12-15 23:57:15,444 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.8 2021-12-15 23:57:15,460 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:15,464 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:15,464 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:23,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.80 % 2021-12-15 23:57:23,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:23,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:25,480 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.80 % 2021-12-15 23:57:25,480 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 17.8 2021-12-15 23:57:25,494 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:25,496 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:25,496 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:33,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.16 % 2021-12-15 23:57:33,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:33,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:35,510 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.16 % 2021-12-15 23:57:35,510 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.16 2021-12-15 23:57:35,526 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:35,529 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:35,529 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:43,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:57:43,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:43,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:45,542 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:57:45,542 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-15 23:57:45,561 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:45,562 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:45,562 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:53,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:57:53,165 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:57:53,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:57:55,575 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:57:55,575 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-15 23:57:55,592 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:57:55,595 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:57:55,595 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:03,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:58:03,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:03,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:05,608 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:58:05,608 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-15 23:58:05,627 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:05,628 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:05,630 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:13,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:58:13,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:13,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:15,641 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:58:15,642 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-15 23:58:15,665 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:15,667 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:15,667 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:23,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-15 23:58:23,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:23,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:25,680 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.57 % 2021-12-15 23:58:25,680 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.57 2021-12-15 23:58:25,697 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:25,700 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:25,700 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:33,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.57 % 2021-12-15 23:58:33,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:33,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:35,711 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.57 % 2021-12-15 23:58:35,713 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.57 2021-12-15 23:58:35,730 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:35,733 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:35,733 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:43,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.90 % 2021-12-15 23:58:43,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:43,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:45,749 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.90 % 2021-12-15 23:58:45,749 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 18.9 2021-12-15 23:58:45,769 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:45,772 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:45,773 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:53,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.08 % 2021-12-15 23:58:53,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:58:53,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:58:55,785 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.08 % 2021-12-15 23:58:55,785 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.08 2021-12-15 23:58:55,799 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:58:55,802 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:58:55,802 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:03,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.21 % 2021-12-15 23:59:03,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:03,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:05,815 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.21 % 2021-12-15 23:59:05,815 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.21 2021-12-15 23:59:05,836 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:05,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:05,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:13,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.21 % 2021-12-15 23:59:13,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:13,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:15,851 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.36 % 2021-12-15 23:59:15,851 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.36 2021-12-15 23:59:15,865 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:15,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:15,868 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:23,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.54 % 2021-12-15 23:59:23,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:23,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:25,880 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.54 % 2021-12-15 23:59:25,881 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.54 2021-12-15 23:59:25,897 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:25,898 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:25,898 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:33,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.67 % 2021-12-15 23:59:33,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:33,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:35,913 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.67 % 2021-12-15 23:59:35,913 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.67 2021-12-15 23:59:35,930 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:35,934 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:35,934 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:43,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.86 % 2021-12-15 23:59:43,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:43,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:45,946 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.86 % 2021-12-15 23:59:45,947 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.86 2021-12-15 23:59:45,963 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:45,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:45,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:53,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.86 % 2021-12-15 23:59:53,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-15 23:59:53,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-15 23:59:55,980 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.86 % 2021-12-15 23:59:55,980 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.86 2021-12-15 23:59:55,996 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-15 23:59:56,000 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-15 23:59:56,000 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:03,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-16 00:00:03,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:03,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:06,012 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-16 00:00:06,012 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.99 2021-12-16 00:00:06,028 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:06,029 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:06,029 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:13,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-16 00:00:13,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:13,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:16,040 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-16 00:00:16,042 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 19.99 2021-12-16 00:00:16,059 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:16,061 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:16,062 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:23,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.99 % 2021-12-16 00:00:23,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:23,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:26,072 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-16 00:00:26,072 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-16 00:00:26,088 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:26,091 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:26,091 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:33,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-16 00:00:33,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:33,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:36,104 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-16 00:00:36,104 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-16 00:00:36,121 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:36,124 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:36,125 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:43,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.32 % 2021-12-16 00:00:43,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:43,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:46,135 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.45 % 2021-12-16 00:00:46,137 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 20.45 2021-12-16 00:00:46,151 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:46,154 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:46,154 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:53,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.45 % 2021-12-16 00:00:53,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:00:53,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:00:56,165 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.63 % 2021-12-16 00:00:56,165 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 20.63 2021-12-16 00:00:56,191 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:00:56,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:00:56,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:03,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.77 % 2021-12-16 00:01:03,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:03,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:06,207 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.91 % 2021-12-16 00:01:06,207 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 20.91 2021-12-16 00:01:06,224 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:06,227 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:06,227 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:13,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:13,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:13,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:16,240 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:16,240 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.09 2021-12-16 00:01:16,256 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:16,259 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:16,259 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:23,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:23,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:23,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:26,272 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:26,272 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.09 2021-12-16 00:01:26,288 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:26,289 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:26,289 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:33,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:33,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:33,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:36,315 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:36,315 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.09 2021-12-16 00:01:36,332 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:36,336 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:36,336 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:43,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:43,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:43,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:46,348 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.09 % 2021-12-16 00:01:46,348 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.09 2021-12-16 00:01:46,371 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:46,375 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:46,375 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:53,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 00:01:53,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:01:53,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:01:56,388 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 00:01:56,390 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.27 2021-12-16 00:01:56,411 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:01:56,414 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:01:56,414 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:03,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.54 % 2021-12-16 00:02:03,724 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:03,724 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:06,433 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.54 % 2021-12-16 00:02:06,434 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.54 2021-12-16 00:02:06,453 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:06,459 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:06,459 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:13,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.69 % 2021-12-16 00:02:13,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:13,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:16,471 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.69 % 2021-12-16 00:02:16,473 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.69 2021-12-16 00:02:16,490 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:16,492 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:16,493 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:23,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.69 % 2021-12-16 00:02:23,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:23,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:26,506 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.69 % 2021-12-16 00:02:26,506 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.69 2021-12-16 00:02:26,523 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:26,526 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:26,526 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:33,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.69 % 2021-12-16 00:02:33,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:33,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:36,539 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.87 % 2021-12-16 00:02:36,539 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.87 2021-12-16 00:02:36,555 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:36,559 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:36,559 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:43,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.87 % 2021-12-16 00:02:43,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:43,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:46,572 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.87 % 2021-12-16 00:02:46,572 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 21.87 2021-12-16 00:02:46,588 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:46,591 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:46,591 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:53,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.01 % 2021-12-16 00:02:53,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:02:53,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:02:56,602 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.19 % 2021-12-16 00:02:56,602 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.19 2021-12-16 00:02:56,621 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:02:56,622 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:02:56,622 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:03,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.46 % 2021-12-16 00:03:03,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:03,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:06,634 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:06,634 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.64 2021-12-16 00:03:06,651 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:06,654 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:06,654 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:13,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:13,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:13,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:16,665 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:16,665 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.64 2021-12-16 00:03:16,683 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:16,686 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:16,687 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:23,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:23,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:23,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:26,697 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:26,697 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.64 2021-12-16 00:03:26,713 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:26,716 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:26,716 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:33,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:33,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:33,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:36,727 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:36,727 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.64 2021-12-16 00:03:36,744 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:36,746 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:36,746 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:43,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:43,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:43,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:46,759 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.64 % 2021-12-16 00:03:46,759 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.64 2021-12-16 00:03:46,775 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:46,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:46,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:53,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.97 % 2021-12-16 00:03:53,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:03:53,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:03:56,789 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.97 % 2021-12-16 00:03:56,789 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.97 2021-12-16 00:03:56,809 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:03:56,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:03:56,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:03,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.97 % 2021-12-16 00:04:03,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:03,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:06,823 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.97 % 2021-12-16 00:04:06,823 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 22.97 2021-12-16 00:04:06,841 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:06,842 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:06,842 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:14,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.24 % 2021-12-16 00:04:14,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:14,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:16,855 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.24 % 2021-12-16 00:04:16,855 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.24 2021-12-16 00:04:16,875 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:16,877 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:16,878 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:24,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.42 % 2021-12-16 00:04:24,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:24,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:26,892 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.42 % 2021-12-16 00:04:26,892 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.42 2021-12-16 00:04:26,921 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:26,923 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:26,924 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:34,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.42 % 2021-12-16 00:04:34,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:34,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:36,936 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.56 % 2021-12-16 00:04:36,936 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.56 2021-12-16 00:04:36,956 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:36,957 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:36,957 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:44,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.56 % 2021-12-16 00:04:44,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:44,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:46,970 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.56 % 2021-12-16 00:04:46,970 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.56 2021-12-16 00:04:46,986 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:46,990 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:46,990 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:54,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.93 % 2021-12-16 00:04:54,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:04:54,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:04:57,002 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.93 % 2021-12-16 00:04:57,002 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.93 2021-12-16 00:04:57,017 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:04:57,019 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:04:57,019 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:04,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.93 % 2021-12-16 00:05:04,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:04,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:07,032 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.93 % 2021-12-16 00:05:07,032 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 23.93 2021-12-16 00:05:07,049 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:07,052 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:07,052 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:14,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.06 % 2021-12-16 00:05:14,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:14,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:17,063 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.06 % 2021-12-16 00:05:17,063 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.06 2021-12-16 00:05:17,082 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:17,085 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:17,085 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:24,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.20 % 2021-12-16 00:05:24,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:24,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:27,098 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.20 % 2021-12-16 00:05:27,098 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.2 2021-12-16 00:05:27,115 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:27,118 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:27,118 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:34,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.20 % 2021-12-16 00:05:34,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:34,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:37,130 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.20 % 2021-12-16 00:05:37,130 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.2 2021-12-16 00:05:37,147 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:37,150 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:37,150 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:44,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 00:05:44,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:44,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:47,161 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 00:05:47,161 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 00:05:47,177 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:47,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:47,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:54,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.47 % 2021-12-16 00:05:54,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:05:54,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:05:57,193 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.47 % 2021-12-16 00:05:57,194 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.47 2021-12-16 00:05:57,217 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:05:57,220 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:05:57,220 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:04,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.47 % 2021-12-16 00:06:04,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:04,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:07,234 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.47 % 2021-12-16 00:06:07,234 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.47 2021-12-16 00:06:07,253 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:07,256 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:07,256 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:14,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.61 % 2021-12-16 00:06:14,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:14,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:17,267 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.61 % 2021-12-16 00:06:17,267 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.61 2021-12-16 00:06:17,282 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:17,285 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:17,285 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:24,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.88 % 2021-12-16 00:06:24,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:24,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:27,296 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.88 % 2021-12-16 00:06:27,296 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 24.88 2021-12-16 00:06:27,315 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:27,316 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:27,316 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:34,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.88 % 2021-12-16 00:06:34,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:34,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:37,329 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.02 % 2021-12-16 00:06:37,329 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.02 2021-12-16 00:06:37,348 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:37,351 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:37,351 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:44,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.15 % 2021-12-16 00:06:44,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:44,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:47,365 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.15 % 2021-12-16 00:06:47,365 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.15 2021-12-16 00:06:47,384 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:47,387 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:47,387 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:54,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.15 % 2021-12-16 00:06:54,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:06:54,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:06:57,398 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.15 % 2021-12-16 00:06:57,400 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.15 2021-12-16 00:06:57,418 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:06:57,421 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:06:57,421 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:04,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.15 % 2021-12-16 00:07:04,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:04,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:07,433 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.33 % 2021-12-16 00:07:07,434 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.33 2021-12-16 00:07:07,450 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:07,453 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:07,453 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:14,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.33 % 2021-12-16 00:07:14,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:14,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:17,467 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.33 % 2021-12-16 00:07:17,467 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.33 2021-12-16 00:07:17,483 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:17,484 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:17,484 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:24,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.33 % 2021-12-16 00:07:24,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:24,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:27,500 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.48 % 2021-12-16 00:07:27,500 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.48 2021-12-16 00:07:27,519 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:27,522 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:27,523 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:34,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.61 % 2021-12-16 00:07:34,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:34,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:37,533 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.61 % 2021-12-16 00:07:37,533 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.61 2021-12-16 00:07:37,551 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:37,552 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:37,552 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:44,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.74 % 2021-12-16 00:07:44,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:44,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:47,565 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.74 % 2021-12-16 00:07:47,565 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.74 2021-12-16 00:07:47,585 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:47,588 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:47,589 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:54,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.74 % 2021-12-16 00:07:54,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:07:54,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:07:57,601 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.74 % 2021-12-16 00:07:57,601 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 25.74 2021-12-16 00:07:57,618 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:07:57,621 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:07:57,621 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:04,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.07 % 2021-12-16 00:08:04,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:04,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:07,631 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.25 % 2021-12-16 00:08:07,632 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.25 2021-12-16 00:08:07,648 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:07,650 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:07,651 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:14,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.25 % 2021-12-16 00:08:14,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:14,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:17,661 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.25 % 2021-12-16 00:08:17,663 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.25 2021-12-16 00:08:17,680 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:17,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:17,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:24,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.25 % 2021-12-16 00:08:24,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:24,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:27,696 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.44 % 2021-12-16 00:08:27,697 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.44 2021-12-16 00:08:27,711 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:27,713 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:27,713 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:34,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.44 % 2021-12-16 00:08:34,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:34,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:37,736 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.44 % 2021-12-16 00:08:37,736 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.44 2021-12-16 00:08:37,755 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:37,759 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:37,759 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:44,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.57 % 2021-12-16 00:08:44,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:44,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:47,770 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.57 % 2021-12-16 00:08:47,770 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.57 2021-12-16 00:08:47,788 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:47,789 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:47,789 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:54,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.57 % 2021-12-16 00:08:54,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:08:54,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:08:57,802 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.57 % 2021-12-16 00:08:57,803 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.57 2021-12-16 00:08:57,826 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:08:57,828 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:08:57,829 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:04,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 00:09:04,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:04,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:07,842 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 00:09:07,842 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 00:09:07,858 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:07,862 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:07,862 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:14,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:14,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:14,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:17,875 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:17,877 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.03 2021-12-16 00:09:17,897 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:17,900 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:17,900 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:24,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:24,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:24,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:27,911 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:27,913 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.03 2021-12-16 00:09:27,928 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:27,930 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:27,930 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:34,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:34,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:34,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:37,944 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.03 % 2021-12-16 00:09:37,944 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.03 2021-12-16 00:09:37,966 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:37,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:37,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:44,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.21 % 2021-12-16 00:09:44,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:44,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:47,982 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.21 % 2021-12-16 00:09:47,982 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.21 2021-12-16 00:09:48,000 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:48,002 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:48,002 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:54,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 00:09:54,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:09:54,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:09:58,013 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.49 % 2021-12-16 00:09:58,013 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.49 2021-12-16 00:09:58,029 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:09:58,032 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:09:58,032 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:04,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.62 % 2021-12-16 00:10:04,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:04,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:08,043 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.62 % 2021-12-16 00:10:08,045 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.62 2021-12-16 00:10:08,062 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:08,063 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:08,063 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:14,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.80 % 2021-12-16 00:10:14,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:14,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:18,075 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.80 % 2021-12-16 00:10:18,075 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.8 2021-12-16 00:10:18,095 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:18,099 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:18,099 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:24,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:24,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:24,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:28,125 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:28,127 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.99 2021-12-16 00:10:28,141 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:28,144 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:28,144 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:34,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:34,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:34,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:38,155 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:38,155 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.99 2021-12-16 00:10:38,173 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:38,177 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:38,177 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:44,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:44,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:44,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:48,190 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:48,190 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 27.99 2021-12-16 00:10:48,207 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:48,209 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:48,209 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:54,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.99 % 2021-12-16 00:10:54,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:10:54,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:10:58,223 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.13 % 2021-12-16 00:10:58,223 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.13 2021-12-16 00:10:58,247 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:10:58,252 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:10:58,252 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:04,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.13 % 2021-12-16 00:11:04,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:04,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:08,266 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.13 % 2021-12-16 00:11:08,266 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.13 2021-12-16 00:11:08,282 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:08,286 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:08,286 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:14,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.26 % 2021-12-16 00:11:14,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:14,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:18,299 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.26 % 2021-12-16 00:11:18,299 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.26 2021-12-16 00:11:18,321 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:18,322 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:18,322 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:24,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.58 % 2021-12-16 00:11:24,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:24,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:28,335 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.58 % 2021-12-16 00:11:28,336 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.58 2021-12-16 00:11:28,354 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:28,357 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:28,358 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:34,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 00:11:34,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:34,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:38,369 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.91 % 2021-12-16 00:11:38,371 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.91 2021-12-16 00:11:38,388 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:38,390 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:38,390 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:44,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.91 % 2021-12-16 00:11:44,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:44,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:48,403 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.91 % 2021-12-16 00:11:48,403 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.91 2021-12-16 00:11:48,420 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:48,421 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:48,421 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:54,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.91 % 2021-12-16 00:11:54,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:11:54,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:11:58,434 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.91 % 2021-12-16 00:11:58,434 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 28.91 2021-12-16 00:11:58,454 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:11:58,457 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:11:58,457 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:04,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.09 % 2021-12-16 00:12:04,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:04,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:08,470 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.09 % 2021-12-16 00:12:08,470 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.09 2021-12-16 00:12:08,486 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:08,489 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:08,489 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:15,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.22 % 2021-12-16 00:12:15,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:15,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:18,516 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.22 % 2021-12-16 00:12:18,516 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.22 2021-12-16 00:12:18,539 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:18,542 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:18,542 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:25,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.22 % 2021-12-16 00:12:25,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:25,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:28,555 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.40 % 2021-12-16 00:12:28,555 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.4 2021-12-16 00:12:28,575 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:28,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:28,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:35,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.54 % 2021-12-16 00:12:35,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:35,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:38,592 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.54 % 2021-12-16 00:12:38,592 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.54 2021-12-16 00:12:38,608 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:38,609 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:38,609 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:45,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.54 % 2021-12-16 00:12:45,081 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:45,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:48,624 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.68 % 2021-12-16 00:12:48,625 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.68 2021-12-16 00:12:48,641 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:48,644 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:48,644 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:55,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.68 % 2021-12-16 00:12:55,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:12:55,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:12:58,657 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.68 % 2021-12-16 00:12:58,657 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.68 2021-12-16 00:12:58,674 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:12:58,677 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:12:58,677 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:05,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.68 % 2021-12-16 00:13:05,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:05,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:08,688 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.68 % 2021-12-16 00:13:08,690 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.68 2021-12-16 00:13:08,704 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:08,707 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:08,707 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:15,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-16 00:13:15,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:15,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:18,720 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-16 00:13:18,720 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 29.82 2021-12-16 00:13:18,739 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:18,742 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:18,742 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:25,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 00:13:25,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:25,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:28,755 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 00:13:28,756 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 00:13:28,773 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:28,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:28,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:35,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.32 % 2021-12-16 00:13:35,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:35,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:38,788 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.32 % 2021-12-16 00:13:38,788 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.32 2021-12-16 00:13:38,815 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:38,818 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:38,819 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:45,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.46 % 2021-12-16 00:13:45,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:45,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:48,831 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.46 % 2021-12-16 00:13:48,831 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.46 2021-12-16 00:13:48,846 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:48,848 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:48,848 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:55,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 00:13:55,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:13:55,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:13:58,861 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 00:13:58,862 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.64 2021-12-16 00:13:58,881 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:13:58,884 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:13:58,884 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:05,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 00:14:05,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:05,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:08,894 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 00:14:08,895 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.64 2021-12-16 00:14:08,913 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:08,915 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:08,915 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:15,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 00:14:15,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:15,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:18,927 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.77 % 2021-12-16 00:14:18,927 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.77 2021-12-16 00:14:18,944 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:18,947 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:18,947 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:25,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.77 % 2021-12-16 00:14:25,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:25,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:28,969 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.77 % 2021-12-16 00:14:28,969 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 30.77 2021-12-16 00:14:28,987 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:28,990 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:28,992 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:35,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.10 % 2021-12-16 00:14:35,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:35,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:39,003 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.10 % 2021-12-16 00:14:39,005 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.1 2021-12-16 00:14:39,023 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:39,026 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:39,026 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:45,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.10 % 2021-12-16 00:14:45,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:45,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:49,039 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.23 % 2021-12-16 00:14:49,039 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.23 2021-12-16 00:14:49,055 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:49,058 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:49,058 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:55,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.23 % 2021-12-16 00:14:55,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:14:55,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:14:59,069 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.23 % 2021-12-16 00:14:59,069 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.23 2021-12-16 00:14:59,085 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:14:59,086 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:14:59,086 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:05,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.23 % 2021-12-16 00:15:05,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:05,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:09,101 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.38 % 2021-12-16 00:15:09,101 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.38 2021-12-16 00:15:09,119 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:09,122 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:09,122 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:15,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.51 % 2021-12-16 00:15:15,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:15,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:19,135 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.51 % 2021-12-16 00:15:19,135 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.51 2021-12-16 00:15:19,154 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:19,157 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:19,157 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:25,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.51 % 2021-12-16 00:15:25,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:25,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:29,171 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.69 % 2021-12-16 00:15:29,171 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.69 2021-12-16 00:15:29,190 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:29,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:29,194 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:35,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.69 % 2021-12-16 00:15:35,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:35,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:39,207 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.83 % 2021-12-16 00:15:39,207 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.83 2021-12-16 00:15:39,226 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:39,230 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:39,230 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:45,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.83 % 2021-12-16 00:15:45,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:45,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:49,242 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.96 % 2021-12-16 00:15:49,243 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 31.96 2021-12-16 00:15:49,259 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:49,262 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:49,263 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:55,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.09 % 2021-12-16 00:15:55,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:15:55,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:15:59,275 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.09 % 2021-12-16 00:15:59,276 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.09 2021-12-16 00:15:59,292 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:15:59,295 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:15:59,296 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:05,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.09 % 2021-12-16 00:16:05,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:05,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:09,308 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.09 % 2021-12-16 00:16:09,309 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.09 2021-12-16 00:16:09,325 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:09,326 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:09,328 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:15,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.09 % 2021-12-16 00:16:15,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:15,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:19,341 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.24 % 2021-12-16 00:16:19,341 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.24 2021-12-16 00:16:19,358 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:19,361 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:19,361 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:25,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.24 % 2021-12-16 00:16:25,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:25,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:29,372 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.24 % 2021-12-16 00:16:29,372 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.24 2021-12-16 00:16:29,390 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:29,391 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:29,391 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:35,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.24 % 2021-12-16 00:16:35,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:35,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:39,403 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.42 % 2021-12-16 00:16:39,404 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.42 2021-12-16 00:16:39,421 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:39,423 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:39,424 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:45,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.55 % 2021-12-16 00:16:45,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:45,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:49,436 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.55 % 2021-12-16 00:16:49,436 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.55 2021-12-16 00:16:49,453 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:49,457 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:49,457 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:55,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.55 % 2021-12-16 00:16:55,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:16:55,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:16:59,469 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.70 % 2021-12-16 00:16:59,470 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.7 2021-12-16 00:16:59,486 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:16:59,486 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:16:59,486 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:05,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.70 % 2021-12-16 00:17:05,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:05,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:09,499 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.83 % 2021-12-16 00:17:09,500 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.83 2021-12-16 00:17:09,515 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:09,516 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:09,517 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:15,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.96 % 2021-12-16 00:17:15,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:15,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:19,529 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.96 % 2021-12-16 00:17:19,529 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.96 2021-12-16 00:17:19,545 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:19,546 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:19,546 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:25,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.96 % 2021-12-16 00:17:25,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:25,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:29,559 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.96 % 2021-12-16 00:17:29,559 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 32.96 2021-12-16 00:17:29,576 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:29,578 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:29,578 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:35,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.96 % 2021-12-16 00:17:35,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:35,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:39,591 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.15 % 2021-12-16 00:17:39,591 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.15 2021-12-16 00:17:39,608 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:39,611 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:39,611 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:45,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.28 % 2021-12-16 00:17:45,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:45,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:49,625 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.28 % 2021-12-16 00:17:49,625 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.28 2021-12-16 00:17:49,644 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:49,645 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:49,645 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:55,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.28 % 2021-12-16 00:17:55,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:17:55,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:17:59,657 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.28 % 2021-12-16 00:17:59,657 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.28 2021-12-16 00:17:59,673 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:17:59,674 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:17:59,674 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:05,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.41 % 2021-12-16 00:18:05,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:05,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:09,686 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.55 % 2021-12-16 00:18:09,686 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.55 2021-12-16 00:18:09,701 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:09,704 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:09,704 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:15,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.69 % 2021-12-16 00:18:15,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:15,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:19,716 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.69 % 2021-12-16 00:18:19,716 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.69 2021-12-16 00:18:19,733 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:19,734 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:19,734 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:25,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.69 % 2021-12-16 00:18:25,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:25,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:29,746 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.69 % 2021-12-16 00:18:29,746 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.69 2021-12-16 00:18:29,773 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:29,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:29,776 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:35,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.69 % 2021-12-16 00:18:35,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:35,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:39,789 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.87 % 2021-12-16 00:18:39,789 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 33.87 2021-12-16 00:18:39,805 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:39,808 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:39,808 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:45,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.02 % 2021-12-16 00:18:45,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:45,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:49,819 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.02 % 2021-12-16 00:18:49,819 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.02 2021-12-16 00:18:49,835 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:49,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:49,838 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:55,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.02 % 2021-12-16 00:18:55,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:18:55,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:18:59,851 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.02 % 2021-12-16 00:18:59,851 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.02 2021-12-16 00:18:59,872 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:18:59,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:18:59,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:05,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.02 % 2021-12-16 00:19:05,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:05,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:09,887 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.15 % 2021-12-16 00:19:09,887 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.15 2021-12-16 00:19:09,903 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:19:09,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:19:09,904 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:15,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.29 % 2021-12-16 00:19:15,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:15,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:19,917 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.29 % 2021-12-16 00:19:19,917 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.29 2021-12-16 00:19:19,937 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:19:19,938 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:19:19,938 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:25,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.42 % 2021-12-16 00:19:25,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:25,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:29,948 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.42 % 2021-12-16 00:19:29,950 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.42 2021-12-16 00:19:29,964 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:19:29,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:19:29,967 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:35,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.55 % 2021-12-16 00:19:35,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:35,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:39,976 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.74 % 2021-12-16 00:19:39,977 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.74 2021-12-16 00:19:39,993 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:19:39,994 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:19:39,994 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:45,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.74 % 2021-12-16 00:19:45,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:45,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:50,005 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.74 % 2021-12-16 00:19:50,006 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.74 2021-12-16 00:19:50,020 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:19:50,023 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:19:50,023 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:19:55,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.74 % 2021-12-16 00:19:55,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:19:55,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:00,033 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.74 % 2021-12-16 00:20:00,033 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.74 2021-12-16 00:20:00,049 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:00,052 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:00,052 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:05,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.88 % 2021-12-16 00:20:05,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:05,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:10,065 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.88 % 2021-12-16 00:20:10,065 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 34.88 2021-12-16 00:20:10,082 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:10,085 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:10,085 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:16,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.88 % 2021-12-16 00:20:16,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:16,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:20,098 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.01 % 2021-12-16 00:20:20,098 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.01 2021-12-16 00:20:20,117 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:20,119 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:20,119 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:26,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.01 % 2021-12-16 00:20:26,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:26,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:30,130 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.01 % 2021-12-16 00:20:30,130 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.01 2021-12-16 00:20:30,145 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:30,148 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:30,148 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:36,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.16 % 2021-12-16 00:20:36,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:36,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:40,158 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.47 % 2021-12-16 00:20:40,158 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.47 2021-12-16 00:20:40,177 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:40,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:40,180 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:46,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.47 % 2021-12-16 00:20:46,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:46,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:50,191 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.47 % 2021-12-16 00:20:50,191 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.47 2021-12-16 00:20:50,207 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:20:50,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:20:50,210 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:20:56,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.47 % 2021-12-16 00:20:56,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:20:56,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:00,223 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.47 % 2021-12-16 00:21:00,223 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.47 2021-12-16 00:21:00,239 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:00,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:00,240 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:06,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.61 % 2021-12-16 00:21:06,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:06,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:10,253 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.61 % 2021-12-16 00:21:10,255 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.61 2021-12-16 00:21:10,273 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:10,276 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:10,276 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:16,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.61 % 2021-12-16 00:21:16,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:16,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:20,289 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.61 % 2021-12-16 00:21:20,289 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.61 2021-12-16 00:21:20,312 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:20,313 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:20,313 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:26,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.74 % 2021-12-16 00:21:26,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:26,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:30,328 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.74 % 2021-12-16 00:21:30,329 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 35.74 2021-12-16 00:21:30,346 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:30,348 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:30,349 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:36,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:21:36,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:36,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:40,361 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:21:40,361 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.06 2021-12-16 00:21:40,380 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:40,381 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:40,381 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:46,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:21:46,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:46,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:50,397 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:21:50,397 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.06 2021-12-16 00:21:50,427 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:21:50,430 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:21:50,430 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:21:56,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:21:56,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:21:56,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:00,444 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:22:00,444 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.06 2021-12-16 00:22:00,470 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:00,474 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:00,474 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:06,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:22:06,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:06,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:10,487 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.06 % 2021-12-16 00:22:10,487 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.06 2021-12-16 00:22:10,512 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:10,513 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:10,513 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:16,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.20 % 2021-12-16 00:22:16,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:16,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:20,526 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.20 % 2021-12-16 00:22:20,526 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.2 2021-12-16 00:22:20,543 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:20,548 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:20,548 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:26,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.33 % 2021-12-16 00:22:26,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:26,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:30,558 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.33 % 2021-12-16 00:22:30,559 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.33 2021-12-16 00:22:30,575 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:30,575 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:30,576 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:36,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.52 % 2021-12-16 00:22:36,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:36,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:40,625 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.52 % 2021-12-16 00:22:40,625 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.52 2021-12-16 00:22:40,644 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:40,648 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:40,648 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:46,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.52 % 2021-12-16 00:22:46,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:46,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:50,663 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.52 % 2021-12-16 00:22:50,663 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.52 2021-12-16 00:22:50,680 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:22:50,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:22:50,684 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:22:56,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.52 % 2021-12-16 00:22:56,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:22:56,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:00,694 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.65 % 2021-12-16 00:23:00,696 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.65 2021-12-16 00:23:00,710 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:00,713 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:00,713 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:06,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.79 % 2021-12-16 00:23:06,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:06,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:10,723 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.79 % 2021-12-16 00:23:10,724 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 36.79 2021-12-16 00:23:10,740 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:10,743 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:10,743 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:16,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.94 % 2021-12-16 00:23:16,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:16,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:20,755 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-16 00:23:20,755 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.07 2021-12-16 00:23:20,776 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:20,778 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:20,778 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:26,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-16 00:23:26,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:26,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:30,789 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.07 % 2021-12-16 00:23:30,789 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.07 2021-12-16 00:23:30,806 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:30,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:30,809 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:36,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.20 % 2021-12-16 00:23:36,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:36,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:40,825 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:23:40,825 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 00:23:40,857 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:40,859 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:40,859 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:46,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:23:46,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:46,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:50,872 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:23:50,874 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 00:23:50,892 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:23:50,895 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:23:50,895 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:23:56,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:23:56,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:23:56,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:00,907 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:00,907 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 00:24:00,924 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:00,927 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:00,927 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:06,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:06,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:06,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:10,938 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:10,938 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 00:24:10,954 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:10,956 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:10,957 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:16,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:16,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:16,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:20,967 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:20,969 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 00:24:20,983 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:20,986 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:20,986 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:26,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 00:24:26,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:26,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:30,996 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.47 % 2021-12-16 00:24:30,997 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.47 2021-12-16 00:24:31,013 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:31,013 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:31,015 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:36,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.47 % 2021-12-16 00:24:36,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:36,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:41,035 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.60 % 2021-12-16 00:24:41,035 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.6 2021-12-16 00:24:41,052 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:41,055 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:41,055 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:46,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.60 % 2021-12-16 00:24:46,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:46,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:51,065 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.60 % 2021-12-16 00:24:51,066 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.6 2021-12-16 00:24:51,085 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:24:51,086 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:24:51,088 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:24:56,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:24:56,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:24:56,556 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:01,098 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:25:01,098 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.88 2021-12-16 00:25:01,115 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:01,118 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:01,118 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:06,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:25:06,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:06,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:11,128 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:25:11,128 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.88 2021-12-16 00:25:11,145 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:11,148 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:11,148 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:16,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:25:16,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:16,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:21,161 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.88 % 2021-12-16 00:25:21,161 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 37.88 2021-12-16 00:25:21,180 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:21,183 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:21,183 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:26,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.02 % 2021-12-16 00:25:26,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:26,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:31,194 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-16 00:25:31,196 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.15 2021-12-16 00:25:31,213 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:31,217 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:31,217 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:36,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-16 00:25:36,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:36,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:41,229 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.15 % 2021-12-16 00:25:41,229 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.15 2021-12-16 00:25:41,246 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:41,249 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:41,249 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:46,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.28 % 2021-12-16 00:25:46,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:46,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:51,259 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.43 % 2021-12-16 00:25:51,259 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.43 2021-12-16 00:25:51,279 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:25:51,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:25:51,282 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:25:56,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:25:56,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:25:56,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:01,296 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:01,296 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.57 2021-12-16 00:26:01,312 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:01,315 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:01,315 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:06,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:06,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:06,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:11,328 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:11,328 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.57 2021-12-16 00:26:11,344 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:11,345 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:11,345 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:16,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:16,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:16,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:21,358 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:21,359 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.57 2021-12-16 00:26:21,378 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:21,381 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:21,382 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:26,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:26,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:26,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:31,394 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.57 % 2021-12-16 00:26:31,394 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.57 2021-12-16 00:26:31,411 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:31,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:31,413 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:36,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-16 00:26:36,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:36,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:41,424 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-16 00:26:41,424 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.7 2021-12-16 00:26:41,444 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:41,448 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:41,448 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:46,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.70 % 2021-12-16 00:26:46,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:46,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:51,460 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:26:51,460 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.97 2021-12-16 00:26:51,480 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:26:51,483 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:26:51,483 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:26:56,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:26:56,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:26:56,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:01,494 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:27:01,494 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.97 2021-12-16 00:27:01,515 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:01,517 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:01,517 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:06,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:27:06,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:06,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:11,530 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:27:11,530 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 38.97 2021-12-16 00:27:11,549 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:11,551 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:11,551 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:16,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.97 % 2021-12-16 00:27:16,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:16,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:21,562 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-16 00:27:21,562 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.1 2021-12-16 00:27:21,578 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:21,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:21,579 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:26,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-16 00:27:26,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:26,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:31,591 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.10 % 2021-12-16 00:27:31,592 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.1 2021-12-16 00:27:31,608 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:31,611 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:31,611 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:36,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.38 % 2021-12-16 00:27:36,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:36,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:41,622 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.38 % 2021-12-16 00:27:41,622 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.38 2021-12-16 00:27:41,638 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:41,641 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:41,641 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:46,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:27:46,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:46,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:51,651 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:27:51,651 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.52 2021-12-16 00:27:51,671 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:27:51,674 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:27:51,676 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:27:56,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:27:56,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:27:56,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:01,688 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:28:01,688 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.52 2021-12-16 00:28:01,706 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:01,710 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:01,710 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:06,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:28:06,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:06,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:11,721 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:28:11,723 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.52 2021-12-16 00:28:11,743 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:11,744 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:11,744 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:16,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:28:16,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:16,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:21,756 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.52 % 2021-12-16 00:28:21,756 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.52 2021-12-16 00:28:21,775 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:21,779 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:21,779 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:26,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.80 % 2021-12-16 00:28:26,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:26,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:31,790 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.80 % 2021-12-16 00:28:31,790 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.8 2021-12-16 00:28:31,809 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:31,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:31,812 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:36,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.80 % 2021-12-16 00:28:36,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:36,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:41,823 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.80 % 2021-12-16 00:28:41,823 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 39.8 2021-12-16 00:28:41,842 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:41,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:41,845 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:46,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.94 % 2021-12-16 00:28:46,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:46,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:51,855 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.07 % 2021-12-16 00:28:51,855 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 40.07 2021-12-16 00:28:51,872 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:28:51,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:28:51,875 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:28:57,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.07 % 2021-12-16 00:28:57,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:28:57,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:29:01,887 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.21 % 2021-12-16 00:29:01,887 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 40.21 2021-12-16 00:29:01,910 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:29:01,911 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:29:01,911 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:29:07,012 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.07% in 3256.62800002 seconds 2021-12-16 00:29:07,022 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-16 00:29:07,022 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-16 00:29:07,023 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 00:29:07,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:29:07,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 00:29:07,582 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:29:07,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 00:29:07,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 00:29:07,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 00:29:07,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 00:29:07,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 00:29:07,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 00:29:07,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 00:29:07,686 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 00:29:07,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 00:29:07,744 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['208265 memcached\n'], []) 2021-12-16 00:29:07,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 00:29:07,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 00:29:07,832 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['208265 memcached\n'], []) 2021-12-16 00:29:07,832 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 00:29:07,836 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 00:29:07,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:29:08,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 00:29:08,224 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:29:08,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 00:29:08,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 00:29:08,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 00:29:08,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 00:29:08,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 00:29:08,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 00:29:08,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 00:29:08,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:29:08,346 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 00:29:08,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:29:08,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:08,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:08,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:08,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:08,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:29:08,477 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 00:29:08,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:29:08,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:08,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:29:08,532 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 00:29:08,532 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 00:29:08,535 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 00:29:08,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:29:08,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 00:29:08,970 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:29:08,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 00:29:08,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 00:29:09,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 00:29:09,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 00:29:09,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 00:29:09,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 00:29:09,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 00:29:09,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:29:09,109 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 00:29:09,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:29:09,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:09,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:09,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:09,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:09,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:09,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:09,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:09,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:09,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:09,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:09,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:09,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:09,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:09,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:09,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:09,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:09,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:09,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:09,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:09,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:09,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:10,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:10,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:29:10,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:29:10,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:29:10,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:29:10,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:29:10,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:29:10,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:29:10,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:29:10,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:29:10,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:29:10,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:29:10,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:29:10,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:10,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:10,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:29:11,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:29:11,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:11,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:11,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:29:11,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:29:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:11,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:11,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:29:11,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:29:11,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:11,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:11,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:29:11,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:29:11,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:11,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:11,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 00:29:11,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 00:29:11,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:29:11,484 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 00:29:11,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:29:11,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:11,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:29:11,529 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 00:29:11,529 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 00:29:11,532 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 00:29:11,926 | test | DEBUG | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.35 % 2021-12-16 00:29:11,926 | test | INFO | pool-3-thread-1 | [task:check:385] Rebalance - status: running, progress: 40.35 2021-12-16 00:29:11,948 | infra | DEBUG | pool-3-thread-1 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:29:11,950 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:29:11,950 | infra | DEBUG | pool-3-thread-1 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:29:12,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:29:12,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 00:29:13,017 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:29:13,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 00:29:13,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 00:29:13,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 00:29:13,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 00:29:13,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 00:29:13,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 00:29:13,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 00:29:13,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:29:13,161 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 00:29:13,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:29:13,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:13,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:13,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:13,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:13,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:13,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:13,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:13,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:13,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:13,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:13,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:13,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:13,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:14,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:14,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:29:14,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:29:14,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:29:14,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:29:14,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:29:14,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:29:14,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:29:14,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:29:14,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:29:14,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:29:14,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:29:14,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:29:14,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:29:14,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:29:14,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:14,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:14,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:29:14,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:29:15,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:15,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:15,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:29:15,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:29:15,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:15,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:15,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:29:15,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:29:15,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:29:15,203 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 00:29:15,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:29:15,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:15,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:29:15,236 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 00:29:15,236 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 00:29:15,240 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 00:29:15,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:29:15,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 00:29:15,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:29:15,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 00:29:15,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 00:29:15,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 00:29:15,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 00:29:15,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 00:29:15,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 00:29:15,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 00:29:15,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:29:15,898 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 00:29:15,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:29:15,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:15,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:16,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:29:16,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:16,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:29:16,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:16,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:29:16,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:16,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:29:16,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:16,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:29:16,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:29:16,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:16,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:29:16,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:29:16,674 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 00:29:16,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:29:16,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:29:16,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:29:16,727 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 00:29:16,727 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 00:29:16,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-16 00:29:16,733 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-16 00:29:21,960 | test | ERROR | pool-3-thread-1 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fd43db25852ff8c9932e2a217c1e0f1e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b22567c2dd18895a57a420f87d3b2f33', u'status': u'notRunning'} - rebalance failed 2021-12-16 00:29:22,000 | test | INFO | pool-3-thread-1 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.67: 2021-12-16 00:29:22,000 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643360347L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:29:20.347Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.29391.174>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,504,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.29109.174>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}}.\nRebalance Operation Id = 787abf80acfd0a5e0fcb8a2c765814bf'} 2021-12-16 00:29:22,000 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643360305L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:29:20.305Z', u'text': u'Worker <0.29333.174> (for action {move,\n {86,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.29391.174>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 504,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.29109.174>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}'} 2021-12-16 00:29:22,002 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643360304L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:29:20.304Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 23:28:59.148459 208496 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 00:29:22,002 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643360291L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:29:20.291Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639970779L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:32:50.779Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639970610L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:32:50.610Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639970383L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:32:50.383Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.68','ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.66'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 787abf80acfd0a5e0fcb8a2c765814bf"} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639917618L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:31:57.618Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 149 seconds.'} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730609L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.609Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1215 22:32:21.577992 203578 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639639730593L, u'shortText': u'message', u'serverTime': u'2021-12-15T23:28:50.593Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.6165.169>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,86,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.65:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.5618.169>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}}.\nRebalance Operation Id = a06304b5801ad90f607f24ba9292d3d5'} 2021-12-16 00:29:22,003 | test | ERROR | pool-3-thread-1 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fd43db25852ff8c9932e2a217c1e0f1e', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=b22567c2dd18895a57a420f87d3b2f33', u'status': u'notRunning'} - rebalance failed 2021-12-16 00:30:16,749 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-16 00:30:16,749 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 00:30:16,762 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-16 00:30:16,776 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-16 00:30:16,776 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 00:30:16,785 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:16,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:16,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:18,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:18,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:18,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:20,821 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:20,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:20,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:22,838 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:22,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:22,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:24,857 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:24,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:24,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:26,891 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:26,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:26,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:28,910 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:28,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:28,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:30,927 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:30,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:30,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:32,946 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:32,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:32,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:34,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:34,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:34,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:36,980 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:36,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:36,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:38,997 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:39,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:39,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:41,016 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:41,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:41,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:43,036 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:43,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:43,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:45,056 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:45,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:45,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:47,078 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:47,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:47,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:49,098 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:49,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:49,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:51,118 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:51,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:51,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:53,138 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:53,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:53,145 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:55,157 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:55,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:55,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:57,176 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:57,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:57,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:30:59,194 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:30:59,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:30:59,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:01,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:01,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:01,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:03,230 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:03,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:03,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:05,249 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:05,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:05,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:07,267 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:07,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:07,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:09,286 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:09,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:09,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:11,306 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:11,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:11,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:13,325 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:13,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:13,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:15,342 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:15,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:15,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:17,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:17,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:17,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:19,384 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:19,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:19,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:21,404 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:21,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:21,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:23,424 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:23,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:23,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:25,444 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:25,451 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:25,451 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:27,469 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:27,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:27,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:34,480 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error timed out 2021-12-16 00:31:34,480 | test | ERROR | MainThread | [rest_client:is_ns_server_running:79] Server 172.23.110.64:8091 is unavailable 2021-12-16 00:31:34,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:34,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:41,493 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error timed out 2021-12-16 00:31:41,493 | test | ERROR | MainThread | [rest_client:is_ns_server_running:79] Server 172.23.110.64:8091 is unavailable 2021-12-16 00:31:41,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:41,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:43,515 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:43,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:43,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:45,539 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:45,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:45,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:47,559 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:47,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:47,568 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:49,586 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:49,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:49,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:51,607 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:51,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:51,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:53,624 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:53,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:53,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:55,641 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:55,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:55,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:57,660 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:57,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:57,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:31:59,680 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:31:59,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:31:59,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:01,700 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:01,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:01,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:03,721 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:03,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:03,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:05,742 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:05,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:05,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:07,762 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:07,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:07,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:09,780 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:09,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:09,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:11,799 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:11,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:11,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:13,821 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 00:32:13,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 00:32:13,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 00:32:15,838 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-16 00:32:15,849 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-16 00:32:15,849 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 00:32:15,861 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-16 00:32:15,872 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-16 00:32:15,872 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 00:32:15,884 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-16 00:32:15,885 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 00:32:15,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:32:16,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 00:32:16,328 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:32:16,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 00:32:16,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 00:32:16,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 00:32:16,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 00:32:16,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 00:32:16,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 00:32:16,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 00:32:16,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:32:16,447 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 00:32:16,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:32:16,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:16,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:16,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:16,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:16,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:32:16,607 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 00:32:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:32:16,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:16,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:32:16,657 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 00:32:16,657 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 00:32:16,660 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 00:32:16,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:32:17,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 00:32:17,108 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:32:17,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 00:32:17,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 00:32:17,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 00:32:17,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 00:32:17,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 00:32:17,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 00:32:17,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 00:32:17,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:32:17,276 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 00:32:17,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:32:17,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:17,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:17,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:17,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:17,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:17,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:17,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:17,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:17,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:17,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:17,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:17,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:17,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:17,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:17,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:17,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:17,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:17,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:17,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:18,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:18,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:18,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:18,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:32:18,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:32:18,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:32:18,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:32:18,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:32:18,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:32:18,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:32:18,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:32:18,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:32:18,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:32:18,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:32:18,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:32:18,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:32:18,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:32:18,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:18,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:18,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:32:18,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:32:19,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:19,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:19,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:32:19,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:32:19,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:19,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:19,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:32:19,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:32:19,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:19,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:19,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 00:32:19,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 00:32:19,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:32:19,285 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 00:32:19,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:32:19,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:19,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:32:19,332 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 00:32:19,332 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 00:32:19,335 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 00:32:19,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:32:19,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 00:32:19,802 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:32:19,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 00:32:19,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 00:32:19,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 00:32:19,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 00:32:19,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 00:32:19,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 00:32:19,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 00:32:19,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:32:19,933 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 00:32:19,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:32:19,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:19,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:20,061 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:20,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:20,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:20,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:20,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:20,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:20,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:20,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:20,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:20,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:20,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:20,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:32:20,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 00:32:20,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:32:20,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 00:32:20,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:20,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:20,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:32:20,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 00:32:20,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:32:21,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 00:32:21,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:32:21,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 00:32:21,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:32:21,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 00:32:21,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:32:21,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 00:32:21,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:32:21,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 00:32:21,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:32:21,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 00:32:21,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:21,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:32:21,515 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 00:32:21,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:32:21,543 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 00:32:21,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:32:21,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:21,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:32:21,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 00:32:21,585 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 00:32:21,588 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 00:32:21,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 00:32:21,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 00:32:22,058 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 00:32:22,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 00:32:22,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 00:32:22,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 00:32:22,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 00:32:22,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 00:32:22,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 00:32:22,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 00:32:22,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 00:32:22,184 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 00:32:22,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 00:32:22,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:22,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 00:32:22,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:22,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 00:32:22,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:22,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 00:32:22,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:22,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 00:32:22,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:22,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 00:32:22,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 00:32:22,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:22,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 00:32:22,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 00:32:22,700 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 00:32:22,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 00:32:22,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 00:32:22,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 00:32:22,734 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 00:32:22,734 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 00:32:22,734 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639639970.29 2021-12-16 00:32:22,734 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 40 2021-12-16 00:32:22,770 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639643542.74 2021-12-16 00:32:22,770 | infra | INFO | pool-3-thread-2 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639643542.74' started 2021-12-16 00:32:22,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-16 00:32:22,776 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-16 00:32:22,796 | test | DEBUG | pool-3-thread-2 | [task:start_rebalance:356] Removing node 172.23.110.66:8091 from cluster 2021-12-16 00:32:22,796 | test | DEBUG | pool-3-thread-2 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.66', 'user': 'Administrator'} 2021-12-16 00:32:22,851 | test | DEBUG | pool-3-thread-2 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 00:32:22,854 | test | INFO | pool-3-thread-2 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 0.286202536922 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 9.93505769552 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 2.28375916216 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 1.18673307549 | --- OUT ---> | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 3.19296879346 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-16 00:32:22,864 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:32:22,864 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:32:22,882 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:32:22,885 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:32:22,885 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:32:32,897 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:32:32,898 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:32:32,927 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:32:32,931 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:32:32,931 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:32:42,944 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:32:42,944 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:32:42,960 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:32:42,961 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:32:42,961 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:32:52,974 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:32:52,974 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:32:52,992 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:32:52,993 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:32:52,994 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:03,009 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:33:03,009 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:33:03,032 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:03,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:03,038 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:13,052 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:33:13,052 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:33:13,072 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:13,073 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:13,073 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:23,119 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:33:23,119 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:33:23,142 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:23,145 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:23,147 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:33,157 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 00:33:33,157 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 00:33:33,174 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:33,176 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:33,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:43,190 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.31 % 2021-12-16 00:33:43,190 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.31 2021-12-16 00:33:43,214 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:43,217 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:43,217 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:33:53,230 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.31 % 2021-12-16 00:33:53,230 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.31 2021-12-16 00:33:53,247 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:33:53,250 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:33:53,250 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:03,262 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:03,263 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-16 00:34:03,278 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:03,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:03,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:13,292 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:13,292 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-16 00:34:13,313 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:13,318 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:13,318 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:22,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:22,789 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.57 2021-12-16 00:34:22,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:22,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:34:22,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:23,328 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:23,328 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-16 00:34:23,344 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:23,345 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:23,345 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:32,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:32,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:34:32,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:33,358 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:33,359 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-16 00:34:33,381 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:33,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:33,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:42,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:42,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:34:42,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:43,394 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.57 % 2021-12-16 00:34:43,394 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 0.57 2021-12-16 00:34:43,411 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:43,414 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:43,414 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:52,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:34:52,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:34:52,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:34:53,426 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:34:53,426 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 1.14 2021-12-16 00:34:53,441 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:34:53,444 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:34:53,444 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:02,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:02,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:02,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:03,457 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:03,457 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 1.14 2021-12-16 00:35:03,476 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:03,480 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:03,480 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:12,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:12,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:12,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:13,493 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:13,493 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 1.14 2021-12-16 00:35:13,513 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:13,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:13,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:22,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:22,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:22,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:23,529 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.14 % 2021-12-16 00:35:23,529 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 1.14 2021-12-16 00:35:23,546 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:23,549 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:23,549 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:32,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.82 % 2021-12-16 00:35:32,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:32,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:33,563 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.82 % 2021-12-16 00:35:33,565 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 1.82 2021-12-16 00:35:33,582 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:33,585 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:33,585 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:42,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.01 % 2021-12-16 00:35:42,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:42,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:43,599 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.01 % 2021-12-16 00:35:43,599 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.01 2021-12-16 00:35:43,618 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:43,619 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:43,619 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:52,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.01 % 2021-12-16 00:35:52,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:35:52,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:35:53,631 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.01 % 2021-12-16 00:35:53,631 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.01 2021-12-16 00:35:53,651 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:35:53,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:35:53,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:02,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:03,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:03,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:03,664 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:03,664 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 00:36:03,680 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:03,684 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:03,684 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:13,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:13,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:13,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:13,697 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:13,697 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 00:36:13,713 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:13,714 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:13,714 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:23,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:23,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:23,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:23,726 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:23,726 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 00:36:23,744 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:23,747 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:23,749 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:33,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:33,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:33,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:33,757 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:33,759 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 00:36:33,779 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:33,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:33,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:43,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:43,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:43,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:43,793 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:43,793 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 00:36:43,819 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:43,822 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:43,822 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:53,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 00:36:53,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:36:53,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:36:53,836 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.84 % 2021-12-16 00:36:53,836 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.84 2021-12-16 00:36:53,852 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:36:53,855 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:36:53,855 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:03,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.84 % 2021-12-16 00:37:03,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:03,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:03,868 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.84 % 2021-12-16 00:37:03,868 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.84 2021-12-16 00:37:03,885 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:03,888 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:03,888 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:13,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.84 % 2021-12-16 00:37:13,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:13,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:13,898 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.84 % 2021-12-16 00:37:13,900 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 2.84 2021-12-16 00:37:13,921 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:13,924 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:13,924 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:23,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:23,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:23,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:23,936 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:23,936 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.15 2021-12-16 00:37:23,954 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:23,957 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:23,957 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:33,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:33,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:33,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:33,970 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:33,970 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.15 2021-12-16 00:37:33,990 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:33,993 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:33,993 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:43,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:43,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:43,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:44,003 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.15 % 2021-12-16 00:37:44,003 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.15 2021-12-16 00:37:44,020 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:44,023 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:44,023 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:53,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.46 % 2021-12-16 00:37:53,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:37:53,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:37:54,036 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.46 % 2021-12-16 00:37:54,036 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.46 2021-12-16 00:37:54,056 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:37:54,059 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:37:54,059 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:03,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.46 % 2021-12-16 00:38:03,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:03,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:04,071 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.46 % 2021-12-16 00:38:04,072 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.46 2021-12-16 00:38:04,089 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:04,094 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:04,094 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:13,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:13,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:13,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:14,107 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:14,108 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-16 00:38:14,131 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:14,134 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:14,134 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:23,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:23,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:23,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:24,148 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:24,148 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-16 00:38:24,174 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:24,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:24,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:33,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:33,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:33,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:34,188 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:34,190 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-16 00:38:34,209 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:34,210 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:34,210 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:43,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:43,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:43,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:44,223 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 00:38:44,223 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 3.71 2021-12-16 00:38:44,243 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:44,246 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:44,246 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:53,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.46 % 2021-12-16 00:38:53,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:38:53,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:38:54,256 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.46 % 2021-12-16 00:38:54,256 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 4.46 2021-12-16 00:38:54,276 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:38:54,278 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:38:54,278 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:03,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.46 % 2021-12-16 00:39:03,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:03,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:04,290 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.46 % 2021-12-16 00:39:04,290 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 4.46 2021-12-16 00:39:04,308 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:04,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:04,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:13,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.77 % 2021-12-16 00:39:13,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:13,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:14,322 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.77 % 2021-12-16 00:39:14,322 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 4.77 2021-12-16 00:39:14,345 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:14,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:14,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:23,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.03 % 2021-12-16 00:39:23,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:23,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:24,359 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.03 % 2021-12-16 00:39:24,361 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.03 2021-12-16 00:39:24,381 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:24,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:24,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:33,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.03 % 2021-12-16 00:39:33,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:33,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:34,397 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.03 % 2021-12-16 00:39:34,397 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.03 2021-12-16 00:39:34,421 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:34,424 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:34,424 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:43,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:39:43,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:43,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:44,436 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:39:44,436 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.34 2021-12-16 00:39:44,454 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:44,459 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:44,459 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:53,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:39:53,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:39:53,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:39:54,470 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:39:54,471 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.34 2021-12-16 00:39:54,490 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:39:54,493 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:39:54,493 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:03,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:03,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:03,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:04,506 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:04,506 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.34 2021-12-16 00:40:04,523 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:04,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:04,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:13,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:13,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:13,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:14,540 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:14,540 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.34 2021-12-16 00:40:14,565 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:14,566 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:14,566 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:23,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:23,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:23,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:24,581 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.34 % 2021-12-16 00:40:24,581 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.34 2021-12-16 00:40:24,602 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:24,605 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:24,605 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:33,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:33,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:33,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:34,617 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:34,617 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.6 2021-12-16 00:40:34,631 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:34,632 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:34,632 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:43,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:43,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:43,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:44,644 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:44,644 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.6 2021-12-16 00:40:44,661 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:44,664 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:44,665 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:53,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:53,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:40:53,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:40:54,676 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.60 % 2021-12-16 00:40:54,676 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.6 2021-12-16 00:40:54,691 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:40:54,694 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:40:54,694 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:03,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:03,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:03,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:04,709 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:04,710 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.91 2021-12-16 00:41:04,732 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:04,733 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:04,733 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:13,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:13,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:13,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:14,746 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:14,746 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.91 2021-12-16 00:41:14,766 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:14,769 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:14,769 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:23,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:23,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:23,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:24,780 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.91 % 2021-12-16 00:41:24,782 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 5.91 2021-12-16 00:41:24,799 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:24,799 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:24,801 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:33,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.22 % 2021-12-16 00:41:33,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:33,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:34,811 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.22 % 2021-12-16 00:41:34,812 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 6.22 2021-12-16 00:41:34,832 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:34,835 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:34,835 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:43,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:41:43,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:43,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:44,848 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:41:44,848 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 6.48 2021-12-16 00:41:44,865 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:44,868 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:44,868 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:53,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:41:53,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:41:53,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:41:54,880 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:41:54,881 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 6.48 2021-12-16 00:41:54,900 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:41:54,901 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:41:54,903 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:03,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:42:03,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:03,724 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:04,913 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.48 % 2021-12-16 00:42:04,913 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 6.48 2021-12-16 00:42:04,941 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:04,944 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:04,946 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:13,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.85 % 2021-12-16 00:42:13,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:13,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:14,957 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-16 00:42:14,957 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-16 00:42:14,977 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:14,980 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:14,980 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:23,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-16 00:42:23,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:23,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:24,992 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-16 00:42:24,992 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-16 00:42:25,007 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:25,010 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:25,010 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:33,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-16 00:42:33,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:33,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:35,022 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.23 % 2021-12-16 00:42:35,022 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.23 2021-12-16 00:42:35,039 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:35,040 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:35,042 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:43,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-16 00:42:43,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:43,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:45,052 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-16 00:42:45,052 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.49 2021-12-16 00:42:45,069 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:45,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:45,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:53,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-16 00:42:53,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:42:53,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:42:55,084 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.49 % 2021-12-16 00:42:55,084 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.49 2021-12-16 00:42:55,101 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:42:55,102 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:42:55,102 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:03,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 00:43:03,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:03,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:05,117 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 00:43:05,117 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.8 2021-12-16 00:43:05,135 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:05,138 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:05,138 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:13,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 00:43:13,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:13,857 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:15,151 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 00:43:15,151 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 7.8 2021-12-16 00:43:15,171 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:15,173 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:15,173 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:23,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.11 % 2021-12-16 00:43:23,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:23,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:25,190 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.11 % 2021-12-16 00:43:25,190 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.11 2021-12-16 00:43:25,226 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:25,230 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:25,230 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:33,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.11 % 2021-12-16 00:43:33,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:33,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:35,243 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.11 % 2021-12-16 00:43:35,243 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.11 2021-12-16 00:43:35,263 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:35,265 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:35,265 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:43,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:43:43,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:43,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:45,276 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:43:45,276 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:43:45,292 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:45,293 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:45,295 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:53,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:43:53,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:43:53,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:43:55,309 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:43:55,309 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:43:55,326 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:43:55,331 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:43:55,331 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:03,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:03,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:03,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:05,342 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:05,342 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:44:05,364 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:05,367 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:05,368 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:13,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:13,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:13,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:15,380 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:15,380 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:44:15,398 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:15,404 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:15,404 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:23,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:23,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:23,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:25,415 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:25,415 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:44:25,434 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:25,434 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:25,436 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:34,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:34,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:34,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:35,446 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 00:44:35,446 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.36 2021-12-16 00:44:35,463 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:35,466 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:35,466 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:44,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-16 00:44:44,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:44,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:45,477 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.93 % 2021-12-16 00:44:45,477 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.93 2021-12-16 00:44:45,496 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:45,499 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:45,499 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:54,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.93 % 2021-12-16 00:44:54,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:44:54,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:44:55,513 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.93 % 2021-12-16 00:44:55,515 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.93 2021-12-16 00:44:55,533 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:44:55,536 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:44:55,536 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:04,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.93 % 2021-12-16 00:45:04,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:04,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:05,548 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.93 % 2021-12-16 00:45:05,548 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 8.93 2021-12-16 00:45:05,566 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:05,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:05,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:14,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.24 % 2021-12-16 00:45:14,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:14,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:15,578 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.24 % 2021-12-16 00:45:15,579 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 9.24 2021-12-16 00:45:15,599 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:15,601 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:15,601 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:24,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.24 % 2021-12-16 00:45:24,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:24,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:25,614 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.24 % 2021-12-16 00:45:25,614 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 9.24 2021-12-16 00:45:25,630 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:25,632 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:25,634 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:34,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.43 % 2021-12-16 00:45:34,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:34,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:35,645 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.43 % 2021-12-16 00:45:35,645 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 9.43 2021-12-16 00:45:35,664 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:35,668 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:35,668 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:44,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:45:44,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:44,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:45,684 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:45:45,686 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-16 00:45:45,703 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:45,706 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:45,706 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:54,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:45:54,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:45:54,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:45:55,717 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:45:55,719 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-16 00:45:55,734 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:45:55,737 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:45:55,737 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:04,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:04,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:04,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:05,750 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:05,752 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-16 00:46:05,769 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:05,773 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:05,773 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:14,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:14,239 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:14,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:15,788 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:15,789 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-16 00:46:15,812 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:15,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:15,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:24,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:24,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:24,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:25,826 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.25 % 2021-12-16 00:46:25,826 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.25 2021-12-16 00:46:25,844 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:25,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:25,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:34,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-16 00:46:34,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:34,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:35,858 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-16 00:46:35,859 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.56 2021-12-16 00:46:35,878 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:35,881 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:35,881 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:44,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.56 % 2021-12-16 00:46:44,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:44,309 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:45,894 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.82 % 2021-12-16 00:46:45,894 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.82 2021-12-16 00:46:45,917 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:45,920 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:45,920 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:54,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.82 % 2021-12-16 00:46:54,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:46:54,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:46:55,930 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.82 % 2021-12-16 00:46:55,930 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.82 2021-12-16 00:46:55,947 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:46:55,948 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:46:55,948 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:04,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.82 % 2021-12-16 00:47:04,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:04,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:05,960 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.82 % 2021-12-16 00:47:05,960 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 10.82 2021-12-16 00:47:05,974 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:05,976 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:05,976 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:14,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:14,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:14,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:15,987 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:15,989 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.13 2021-12-16 00:47:16,010 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:16,013 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:16,015 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:24,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:24,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:24,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:26,026 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:26,026 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.13 2021-12-16 00:47:26,043 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:26,046 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:26,046 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:34,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:34,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:34,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:36,059 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:36,059 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.13 2021-12-16 00:47:36,073 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:36,076 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:36,076 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:44,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:44,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:44,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:46,088 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.13 % 2021-12-16 00:47:46,088 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.13 2021-12-16 00:47:46,109 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:46,112 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:46,112 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:54,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:47:54,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:47:54,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:47:56,125 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:47:56,125 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.39 2021-12-16 00:47:56,141 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:47:56,144 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:47:56,144 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:04,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:48:04,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:04,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:06,157 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:48:06,157 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.39 2021-12-16 00:48:06,177 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:06,178 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:06,180 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:14,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:48:14,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:14,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:16,191 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.39 % 2021-12-16 00:48:16,191 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.39 2021-12-16 00:48:16,210 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:16,213 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:16,213 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:24,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:24,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:24,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:26,223 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:26,223 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.7 2021-12-16 00:48:26,240 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:26,242 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:26,242 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:34,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:34,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:34,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:36,253 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:36,253 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.7 2021-12-16 00:48:36,273 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:36,278 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:36,278 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:44,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:44,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:44,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:46,289 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.70 % 2021-12-16 00:48:46,289 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 11.7 2021-12-16 00:48:46,308 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:46,311 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:46,311 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:54,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.07 % 2021-12-16 00:48:54,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:48:54,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:48:56,326 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.57 % 2021-12-16 00:48:56,326 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 12.57 2021-12-16 00:48:56,345 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:48:56,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:48:56,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:04,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:04,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:04,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:06,361 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:06,361 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:06,381 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:06,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:06,384 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:14,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:14,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:14,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:16,394 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:16,394 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:16,414 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:16,417 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:16,417 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:24,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:24,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:24,609 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:26,427 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:26,427 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:26,444 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:26,447 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:26,447 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:34,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:34,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:34,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:36,460 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:36,460 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:36,477 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:36,480 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:36,482 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:44,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:44,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:44,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:46,493 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:46,493 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:46,513 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:46,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:46,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:54,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:54,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:49:54,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:49:56,529 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.02 % 2021-12-16 00:49:56,529 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.02 2021-12-16 00:49:56,549 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:49:56,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:49:56,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:04,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 00:50:04,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:04,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:06,566 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 00:50:06,566 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.27 2021-12-16 00:50:06,584 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:06,586 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:06,588 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:14,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:14,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:14,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:16,601 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:16,601 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.58 2021-12-16 00:50:16,618 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:16,619 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:16,619 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:24,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:24,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:24,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:26,631 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:26,631 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.58 2021-12-16 00:50:26,654 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:26,655 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:26,655 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:34,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:34,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:34,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:36,667 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:36,667 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.58 2021-12-16 00:50:36,690 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:36,691 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:36,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:44,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.58 % 2021-12-16 00:50:44,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:44,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:46,703 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.89 % 2021-12-16 00:50:46,703 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.89 2021-12-16 00:50:46,719 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:46,721 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:46,721 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:54,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.89 % 2021-12-16 00:50:54,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:50:54,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:50:56,733 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.89 % 2021-12-16 00:50:56,733 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 13.89 2021-12-16 00:50:56,756 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:50:56,757 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:50:56,759 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:04,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:04,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:04,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:06,769 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:06,769 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:06,788 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:06,790 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:06,790 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:14,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:14,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:14,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:16,802 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:16,802 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:16,818 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:16,819 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:16,819 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:24,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:24,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:24,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:26,832 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:26,832 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:26,855 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:26,857 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:26,857 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:34,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:34,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:34,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:36,872 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:36,874 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:36,892 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:36,897 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:36,898 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:44,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:44,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:44,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:46,911 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:46,911 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:46,933 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:46,937 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:46,937 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:54,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:54,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:51:54,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:51:56,950 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.15 % 2021-12-16 00:51:56,950 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.15 2021-12-16 00:51:56,967 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:51:56,970 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:51:56,970 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:04,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.72 % 2021-12-16 00:52:04,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:04,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:06,982 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.72 % 2021-12-16 00:52:06,982 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.72 2021-12-16 00:52:07,002 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:07,003 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:07,003 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:14,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.72 % 2021-12-16 00:52:14,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:14,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:17,015 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.72 % 2021-12-16 00:52:17,015 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 14.72 2021-12-16 00:52:17,033 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:17,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:17,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:24,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.10 % 2021-12-16 00:52:24,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:24,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:27,068 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.28 % 2021-12-16 00:52:27,068 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 15.28 2021-12-16 00:52:27,085 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:27,088 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:27,088 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:35,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.47 % 2021-12-16 00:52:35,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:35,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:37,099 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.47 % 2021-12-16 00:52:37,101 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 15.47 2021-12-16 00:52:37,119 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:37,124 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:37,124 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:45,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:52:45,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:45,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:47,137 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:52:47,137 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 15.78 2021-12-16 00:52:47,155 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:47,160 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:47,160 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:55,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:52:55,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:52:55,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:52:57,171 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:52:57,171 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 15.78 2021-12-16 00:52:57,193 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:52:57,194 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:52:57,194 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:05,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:53:05,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:05,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:07,206 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.78 % 2021-12-16 00:53:07,207 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 15.78 2021-12-16 00:53:07,223 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:07,226 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:07,226 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:15,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:15,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:15,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:17,237 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:17,237 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.04 2021-12-16 00:53:17,256 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:17,259 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:17,259 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:25,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:25,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:25,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:27,278 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:27,280 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.04 2021-12-16 00:53:27,299 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:27,302 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:27,302 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:35,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:35,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:35,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:37,315 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:37,315 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.04 2021-12-16 00:53:37,332 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:37,335 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:37,335 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:45,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:45,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:45,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:47,345 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:47,345 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.04 2021-12-16 00:53:47,375 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:47,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:47,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:55,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.04 % 2021-12-16 00:53:55,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:53:55,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:53:57,391 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 00:53:57,391 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.35 2021-12-16 00:53:57,410 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:53:57,413 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:53:57,413 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:05,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 00:54:05,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:05,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:07,423 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 00:54:07,423 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.35 2021-12-16 00:54:07,440 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:07,443 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:07,443 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:15,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 00:54:15,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:15,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:17,453 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 00:54:17,453 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.61 2021-12-16 00:54:17,470 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:17,470 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:17,471 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:25,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 00:54:25,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:25,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:27,484 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 00:54:27,484 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.61 2021-12-16 00:54:27,500 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:27,503 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:27,503 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:35,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:35,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:35,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:37,513 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:37,515 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-16 00:54:37,532 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:37,535 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:37,536 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:45,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:45,278 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:45,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:47,546 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:47,546 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-16 00:54:47,562 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:47,565 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:47,565 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:55,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:55,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:54:55,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:54:57,588 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:54:57,588 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-16 00:54:57,605 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:54:57,607 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:54:57,608 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:05,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:55:05,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:05,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:07,618 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:55:07,619 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-16 00:55:07,635 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:07,638 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:07,638 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:15,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:55:15,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:15,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:17,651 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.92 % 2021-12-16 00:55:17,651 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 16.92 2021-12-16 00:55:17,667 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:17,671 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:17,671 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:25,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 00:55:25,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:25,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:27,684 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 00:55:27,684 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 17.17 2021-12-16 00:55:27,703 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:27,707 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:27,707 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:35,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.55 % 2021-12-16 00:55:35,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:35,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:37,719 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.55 % 2021-12-16 00:55:37,719 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 17.55 2021-12-16 00:55:37,737 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:37,740 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:37,740 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:45,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.92 % 2021-12-16 00:55:45,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:45,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:47,760 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.92 % 2021-12-16 00:55:47,762 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 17.92 2021-12-16 00:55:47,780 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:47,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:47,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:55,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:55:55,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:55:55,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:55:57,799 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:55:57,799 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.23 2021-12-16 00:55:57,815 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:55:57,818 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:55:57,819 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:05,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:56:05,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:05,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:07,854 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:56:07,854 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.23 2021-12-16 00:56:07,875 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:07,878 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:07,878 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:15,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:56:15,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:15,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:17,892 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.23 % 2021-12-16 00:56:17,892 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.23 2021-12-16 00:56:17,914 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:17,915 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:17,915 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:25,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:25,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:25,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:27,933 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:27,934 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:56:27,950 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:27,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:27,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:35,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:35,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:35,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:37,967 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:37,967 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:56:37,987 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:37,990 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:37,990 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:45,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:45,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:45,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:48,016 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:48,016 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:56:48,035 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:48,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:48,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:55,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:55,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:56:55,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:56:58,051 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:56:58,051 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:56:58,069 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:56:58,073 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:56:58,073 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:05,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:57:05,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:05,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:08,086 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:57:08,086 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:57:08,108 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:08,111 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:08,111 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:15,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:57:15,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:15,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:18,125 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.80 % 2021-12-16 00:57:18,125 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 18.8 2021-12-16 00:57:18,140 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:18,141 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:18,141 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:25,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.06 % 2021-12-16 00:57:25,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:25,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:28,154 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.06 % 2021-12-16 00:57:28,154 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.06 2021-12-16 00:57:28,170 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:28,171 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:28,171 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:35,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.06 % 2021-12-16 00:57:35,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:35,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:38,184 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.06 % 2021-12-16 00:57:38,184 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.06 2021-12-16 00:57:38,201 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:38,204 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:38,204 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:45,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:57:45,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:45,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:48,217 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:57:48,217 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.37 2021-12-16 00:57:48,233 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:48,234 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:48,234 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:55,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:57:55,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:57:55,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:57:58,250 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:57:58,250 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.37 2021-12-16 00:57:58,269 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:57:58,273 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:57:58,273 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:05,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:58:05,703 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:05,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:08,286 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.37 % 2021-12-16 00:58:08,286 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.37 2021-12-16 00:58:08,305 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:08,308 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:08,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:15,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:15,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:15,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:18,322 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:18,322 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.94 2021-12-16 00:58:18,338 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:18,341 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:18,341 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:25,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:25,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:25,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:28,358 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:28,358 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.94 2021-12-16 00:58:28,380 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:28,382 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:28,382 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:35,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:35,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:35,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:38,394 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:38,394 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 19.94 2021-12-16 00:58:38,420 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:38,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:38,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:45,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.94 % 2021-12-16 00:58:45,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:45,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:48,436 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.13 % 2021-12-16 00:58:48,436 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 20.13 2021-12-16 00:58:48,453 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:48,457 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:48,457 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:55,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:58:55,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:58:55,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:58:58,471 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:58:58,471 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 20.69 2021-12-16 00:58:58,487 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:58:58,490 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:58:58,490 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:05,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:59:05,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:05,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:08,505 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:59:08,505 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 20.69 2021-12-16 00:59:08,526 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:08,529 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:08,529 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:15,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:59:15,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:15,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:18,542 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.69 % 2021-12-16 00:59:18,542 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 20.69 2021-12-16 00:59:18,559 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:18,562 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:18,562 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:25,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:25,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:25,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:28,573 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:28,573 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.26 2021-12-16 00:59:28,592 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:28,594 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:28,594 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:35,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:35,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:35,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:38,608 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:38,608 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.26 2021-12-16 00:59:38,625 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:38,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:38,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:45,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:45,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:45,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:48,650 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:48,650 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.26 2021-12-16 00:59:48,670 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:48,674 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:48,674 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:55,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:55,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 00:59:55,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 00:59:58,687 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.26 % 2021-12-16 00:59:58,688 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.26 2021-12-16 00:59:58,707 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 00:59:58,710 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 00:59:58,710 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:05,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:05,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:05,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:08,723 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:08,723 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.57 2021-12-16 01:00:08,740 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:08,742 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:08,742 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:15,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:15,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:15,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:18,756 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:18,756 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.57 2021-12-16 01:00:18,779 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:18,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:18,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:25,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:25,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:25,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:28,796 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.57 % 2021-12-16 01:00:28,796 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.57 2021-12-16 01:00:28,812 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:28,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:28,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:36,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:36,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:36,010 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:38,828 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:38,828 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.83 2021-12-16 01:00:38,851 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:38,854 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:38,854 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:46,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:46,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:46,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:48,869 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:48,869 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.83 2021-12-16 01:00:48,888 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:48,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:48,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:56,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:56,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:00:56,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:00:58,904 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:00:58,904 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.83 2021-12-16 01:00:58,920 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:00:58,923 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:00:58,923 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:06,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:01:06,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:06,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:08,940 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.83 % 2021-12-16 01:01:08,940 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 21.83 2021-12-16 01:01:08,964 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:08,967 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:08,967 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:16,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:16,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:16,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:18,980 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:18,980 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.14 2021-12-16 01:01:19,003 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:19,007 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:19,009 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:26,117 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:26,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:26,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:29,023 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:29,023 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.14 2021-12-16 01:01:29,039 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:29,042 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:29,042 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:36,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:36,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:36,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:39,055 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.14 % 2021-12-16 01:01:39,056 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.14 2021-12-16 01:01:39,075 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:39,079 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:39,079 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:46,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.39 % 2021-12-16 01:01:46,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:46,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:49,095 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.70 % 2021-12-16 01:01:49,095 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.7 2021-12-16 01:01:49,117 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:49,119 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:49,119 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:56,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.70 % 2021-12-16 01:01:56,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:01:56,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:01:59,141 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.70 % 2021-12-16 01:01:59,141 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.7 2021-12-16 01:01:59,160 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:01:59,163 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:01:59,163 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:06,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.70 % 2021-12-16 01:02:06,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:06,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:09,176 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.70 % 2021-12-16 01:02:09,176 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 22.7 2021-12-16 01:02:09,196 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:09,201 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:09,203 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:16,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.08 % 2021-12-16 01:02:16,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:16,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:19,214 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:19,216 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 23.45 2021-12-16 01:02:19,232 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:19,233 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:19,233 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:26,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:26,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:26,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:29,246 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:29,246 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 23.45 2021-12-16 01:02:29,266 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:29,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:29,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:36,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:36,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:36,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:39,286 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:39,286 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 23.45 2021-12-16 01:02:39,306 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:39,312 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:39,313 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:46,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.45 % 2021-12-16 01:02:46,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:46,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:49,326 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.71 % 2021-12-16 01:02:49,326 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 23.71 2021-12-16 01:02:49,346 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:49,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:49,348 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:56,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:02:56,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:02:56,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:02:59,361 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:02:59,361 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.02 2021-12-16 01:02:59,381 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:02:59,385 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:02:59,385 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:06,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:06,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:06,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:09,398 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:09,398 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.02 2021-12-16 01:03:09,418 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:09,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:09,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:16,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:16,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:16,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:19,434 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:19,434 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.02 2021-12-16 01:03:19,450 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:19,453 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:19,453 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:26,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:26,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:26,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:29,464 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.02 % 2021-12-16 01:03:29,464 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.02 2021-12-16 01:03:29,483 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:29,484 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:29,486 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:36,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.33 % 2021-12-16 01:03:36,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:36,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:39,499 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.33 % 2021-12-16 01:03:39,499 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.33 2021-12-16 01:03:39,525 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:39,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:39,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:46,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.33 % 2021-12-16 01:03:46,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:46,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:49,543 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.33 % 2021-12-16 01:03:49,543 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.33 2021-12-16 01:03:49,565 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:49,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:49,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:56,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:03:56,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:03:56,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:03:59,582 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:03:59,582 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.59 2021-12-16 01:03:59,598 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:03:59,601 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:03:59,601 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:06,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:06,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:06,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:09,615 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:09,615 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.59 2021-12-16 01:04:09,631 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:09,634 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:09,635 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:16,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:16,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:16,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:19,648 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:19,650 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.59 2021-12-16 01:04:19,667 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:19,670 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:19,671 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:26,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:26,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:26,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:29,684 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:29,684 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.59 2021-12-16 01:04:29,700 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:29,703 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:29,703 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:36,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:36,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:36,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:39,719 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 01:04:39,720 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.59 2021-12-16 01:04:39,743 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:39,747 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:39,747 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:46,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.90 % 2021-12-16 01:04:46,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:46,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:49,763 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.90 % 2021-12-16 01:04:49,765 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.9 2021-12-16 01:04:49,779 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:49,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:49,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:56,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.90 % 2021-12-16 01:04:56,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:04:56,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:04:59,796 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.90 % 2021-12-16 01:04:59,796 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 24.9 2021-12-16 01:04:59,821 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:04:59,823 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:04:59,823 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:06,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.90 % 2021-12-16 01:05:06,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:06,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:09,836 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.16 % 2021-12-16 01:05:09,838 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 25.16 2021-12-16 01:05:09,852 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:05:09,855 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:05:09,855 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:16,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.47 % 2021-12-16 01:05:16,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:16,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:19,868 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.47 % 2021-12-16 01:05:19,868 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 25.47 2021-12-16 01:05:19,888 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:05:19,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:05:19,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:26,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.66 % 2021-12-16 01:05:26,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:26,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:29,904 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.84 % 2021-12-16 01:05:29,904 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 25.84 2021-12-16 01:05:29,921 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:05:29,923 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:05:29,923 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:36,676 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:05:36,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:36,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:39,937 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:05:39,937 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.22 2021-12-16 01:05:39,960 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:05:39,963 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:05:39,963 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:46,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:05:46,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:46,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:49,977 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:05:49,977 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.22 2021-12-16 01:05:49,993 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:05:49,996 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:05:49,996 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:05:56,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:05:56,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:05:56,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:00,013 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:06:00,013 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.22 2021-12-16 01:06:00,032 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:00,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:00,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:06,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:06:06,747 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:06,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:10,049 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:06:10,049 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.22 2021-12-16 01:06:10,066 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:10,069 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:10,069 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:16,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:06:16,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:16,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:20,082 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.22 % 2021-12-16 01:06:20,082 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.22 2021-12-16 01:06:20,105 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:20,105 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:20,107 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:26,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.53 % 2021-12-16 01:06:26,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:26,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:30,122 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:30,124 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.79 2021-12-16 01:06:30,148 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:30,151 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:30,153 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:36,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:36,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:36,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:40,167 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:40,167 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.79 2021-12-16 01:06:40,184 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:40,187 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:40,187 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:46,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:46,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:46,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:50,200 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:50,200 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 26.79 2021-12-16 01:06:50,223 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:06:50,224 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:06:50,226 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:06:56,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.79 % 2021-12-16 01:06:56,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:06:56,864 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:00,240 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:00,240 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.1 2021-12-16 01:07:00,263 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:00,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:00,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:06,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:06,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:06,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:10,283 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:10,283 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.1 2021-12-16 01:07:10,311 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:10,312 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:10,312 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:16,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:16,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:16,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:20,325 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:20,325 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.1 2021-12-16 01:07:20,342 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:20,345 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:20,345 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:26,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:26,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:26,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:30,355 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:30,355 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.1 2021-12-16 01:07:30,374 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:30,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:30,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:36,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:36,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:36,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:40,388 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.10 % 2021-12-16 01:07:40,388 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.1 2021-12-16 01:07:40,405 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:40,407 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:40,408 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:46,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:07:46,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:46,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:50,421 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:07:50,421 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.35 2021-12-16 01:07:50,440 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:07:50,446 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:07:50,446 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:07:56,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:07:57,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:07:57,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:00,459 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:08:00,459 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.35 2021-12-16 01:08:00,477 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:00,480 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:00,480 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:07,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:08:07,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:07,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:10,494 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.35 % 2021-12-16 01:08:10,494 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.35 2021-12-16 01:08:10,517 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:10,522 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:10,522 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:17,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.66 % 2021-12-16 01:08:17,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:17,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:20,533 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.66 % 2021-12-16 01:08:20,533 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.66 2021-12-16 01:08:20,549 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:20,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:20,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:27,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.66 % 2021-12-16 01:08:27,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:27,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:30,563 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 01:08:30,563 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 01:08:30,579 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:30,582 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:30,582 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:37,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.41 % 2021-12-16 01:08:37,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:37,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:40,594 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.41 % 2021-12-16 01:08:40,595 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.41 2021-12-16 01:08:40,611 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:40,614 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:40,615 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:47,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.41 % 2021-12-16 01:08:47,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:47,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:50,627 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:08:50,627 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.72 2021-12-16 01:08:50,645 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:08:50,648 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:08:50,648 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:08:57,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:08:57,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:08:57,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:00,660 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:09:00,660 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.72 2021-12-16 01:09:00,677 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:00,678 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:00,678 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:07,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:09:07,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:07,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:10,690 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:09:10,690 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.72 2021-12-16 01:09:10,707 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:10,710 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:10,710 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:17,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.72 % 2021-12-16 01:09:17,186 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:17,186 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:20,723 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:20,723 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.98 2021-12-16 01:09:20,739 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:20,742 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:20,743 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:27,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:27,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:27,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:30,755 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:30,755 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.98 2021-12-16 01:09:30,773 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:30,776 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:30,776 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:37,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:37,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:37,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:40,790 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:40,792 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 28.98 2021-12-16 01:09:40,809 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:40,812 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:40,812 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:47,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.98 % 2021-12-16 01:09:47,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:47,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:50,855 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:09:50,855 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.29 2021-12-16 01:09:50,874 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:09:50,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:09:50,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:09:57,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:09:57,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:09:57,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:00,890 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:10:00,890 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.29 2021-12-16 01:10:00,911 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:00,913 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:00,914 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:07,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:10:07,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:07,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:10,926 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:10:10,926 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.29 2021-12-16 01:10:10,944 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:10,947 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:10,947 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:17,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:10:17,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:17,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:20,966 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.29 % 2021-12-16 01:10:20,966 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.29 2021-12-16 01:10:20,987 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:20,990 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:20,990 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:27,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.55 % 2021-12-16 01:10:27,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:27,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:31,003 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.55 % 2021-12-16 01:10:31,005 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.55 2021-12-16 01:10:31,023 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:31,025 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:31,025 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:37,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:10:37,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:37,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:41,038 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:10:41,039 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.86 2021-12-16 01:10:41,063 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:41,066 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:41,066 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:47,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:10:47,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:47,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:51,082 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:10:51,082 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.86 2021-12-16 01:10:51,099 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:10:51,102 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:10:51,102 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:10:57,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:10:57,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:10:57,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:01,119 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:11:01,119 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.86 2021-12-16 01:11:01,140 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:01,141 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:01,141 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:07,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:11:07,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:07,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:11,154 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:11:11,154 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.86 2021-12-16 01:11:11,173 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:11,176 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:11,176 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:17,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:11:17,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:17,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:21,190 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 01:11:21,190 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 29.86 2021-12-16 01:11:21,207 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:21,209 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:21,210 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:27,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.12 % 2021-12-16 01:11:27,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:27,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:31,223 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.12 % 2021-12-16 01:11:31,223 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 30.12 2021-12-16 01:11:31,247 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:31,250 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:31,250 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:37,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.43 % 2021-12-16 01:11:37,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:37,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:41,263 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.43 % 2021-12-16 01:11:41,263 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 30.43 2021-12-16 01:11:41,283 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:41,286 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:41,286 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:47,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.62 % 2021-12-16 01:11:47,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:47,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:51,301 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.80 % 2021-12-16 01:11:51,302 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 30.8 2021-12-16 01:11:51,318 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:11:51,321 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:11:51,322 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:11:57,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 01:11:57,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:11:57,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:01,332 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 01:12:01,332 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.18 2021-12-16 01:12:01,349 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:01,351 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:01,351 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:07,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 01:12:07,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:07,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:11,368 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 01:12:11,368 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.18 2021-12-16 01:12:11,394 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:11,397 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:11,397 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:17,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 01:12:17,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:17,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:21,408 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:21,408 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:12:21,430 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:21,434 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:21,434 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:27,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:27,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:27,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:31,450 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:31,450 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:12:31,467 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:31,470 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:31,470 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:37,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:37,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:37,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:41,480 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:41,482 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:12:41,500 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:41,503 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:41,503 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:47,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:47,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:47,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:51,516 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:51,516 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:12:51,533 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:12:51,536 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:12:51,538 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:12:57,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:12:57,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:12:57,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:01,552 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:13:01,552 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:13:01,572 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:01,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:01,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:07,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:13:07,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:07,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:11,586 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:13:11,586 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 31.75 2021-12-16 01:13:11,604 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:11,607 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:11,607 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:17,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.75 % 2021-12-16 01:13:17,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:17,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:21,621 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.00 % 2021-12-16 01:13:21,621 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.0 2021-12-16 01:13:21,641 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:21,642 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:21,644 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:27,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:27,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:27,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:31,654 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:31,655 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.31 2021-12-16 01:13:31,671 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:31,674 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:31,674 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:37,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:37,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:37,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:41,684 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:41,686 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.31 2021-12-16 01:13:41,700 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:41,703 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:41,703 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:47,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:47,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:47,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:51,717 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:51,717 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.31 2021-12-16 01:13:51,733 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:13:51,736 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:13:51,736 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:13:57,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:13:57,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:13:57,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:01,753 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:14:01,753 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.31 2021-12-16 01:14:01,769 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:01,773 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:01,773 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:07,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.31 % 2021-12-16 01:14:07,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:07,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:11,786 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.62 % 2021-12-16 01:14:11,786 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.62 2021-12-16 01:14:11,806 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:11,812 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:11,812 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:17,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:17,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:17,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:21,825 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:21,825 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-16 01:14:21,842 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:21,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:21,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:27,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:27,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:27,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:31,868 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:31,868 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-16 01:14:31,888 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:31,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:31,891 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:37,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:37,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:37,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:41,905 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:41,905 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-16 01:14:41,921 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:41,924 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:41,924 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:47,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:47,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:47,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:51,934 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:51,936 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-16 01:14:51,951 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:14:51,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:14:51,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:14:57,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:14:57,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:14:57,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:01,967 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.88 % 2021-12-16 01:15:01,969 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 32.88 2021-12-16 01:15:01,986 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:01,989 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:01,989 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:07,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.07 % 2021-12-16 01:15:07,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:07,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:12,003 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.26 % 2021-12-16 01:15:12,003 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 33.26 2021-12-16 01:15:12,019 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:12,022 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:12,022 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:17,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 01:15:17,974 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:17,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:22,033 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 01:15:22,033 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 33.94 2021-12-16 01:15:22,049 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:22,052 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:22,052 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:27,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:27,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:27,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:32,065 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:32,065 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.2 2021-12-16 01:15:32,082 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:32,085 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:32,085 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:38,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:38,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:38,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:42,098 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:42,098 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.2 2021-12-16 01:15:42,115 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:42,117 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:42,117 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:48,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:48,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:48,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:52,130 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:52,130 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.2 2021-12-16 01:15:52,148 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:15:52,150 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:15:52,150 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:15:58,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.20 % 2021-12-16 01:15:58,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:15:58,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:02,165 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.51 % 2021-12-16 01:16:02,165 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.51 2021-12-16 01:16:02,181 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:02,184 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:02,184 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:08,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.51 % 2021-12-16 01:16:08,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:08,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:12,200 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.51 % 2021-12-16 01:16:12,200 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.51 2021-12-16 01:16:12,217 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:12,220 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:12,220 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:18,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:18,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:18,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:22,234 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:22,234 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:16:22,252 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:22,253 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:22,253 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:28,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:28,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:28,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:32,266 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:32,267 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:16:32,288 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:32,292 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:32,292 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:38,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:38,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:38,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:42,303 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:42,303 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:16:42,321 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:42,323 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:42,323 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:48,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:48,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:48,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:52,338 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:52,338 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:16:52,362 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:16:52,365 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:16:52,365 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:16:58,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:16:58,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:16:58,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:02,377 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:17:02,377 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:17:02,397 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:02,398 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:02,398 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:08,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:17:08,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:08,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:12,413 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.77 % 2021-12-16 01:17:12,413 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 34.77 2021-12-16 01:17:12,430 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:12,433 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:12,434 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:18,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:18,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:18,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:22,451 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:22,453 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.34 2021-12-16 01:17:22,470 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:22,474 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:22,474 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:28,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:28,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:28,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:32,486 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:32,486 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.34 2021-12-16 01:17:32,505 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:32,506 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:32,506 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:38,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:38,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:38,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:42,522 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:42,522 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.34 2021-12-16 01:17:42,539 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:42,542 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:42,542 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:48,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:48,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:48,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:52,555 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:52,555 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.34 2021-12-16 01:17:52,572 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:17:52,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:17:52,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:17:58,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.34 % 2021-12-16 01:17:58,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:17:58,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:02,589 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.65 % 2021-12-16 01:18:02,591 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.65 2021-12-16 01:18:02,608 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:02,611 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:02,611 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:08,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.65 % 2021-12-16 01:18:08,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:08,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:12,625 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.65 % 2021-12-16 01:18:12,625 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.65 2021-12-16 01:18:12,651 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:12,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:12,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:18,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.91 % 2021-12-16 01:18:18,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:18,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:22,667 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.91 % 2021-12-16 01:18:22,667 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 35.91 2021-12-16 01:18:22,683 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:22,684 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:22,686 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:28,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.09 % 2021-12-16 01:18:28,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:28,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:32,697 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.09 % 2021-12-16 01:18:32,697 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 36.09 2021-12-16 01:18:32,713 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:32,717 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:32,717 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:38,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:18:38,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:38,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:42,727 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:18:42,727 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 36.66 2021-12-16 01:18:42,743 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:42,746 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:42,746 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:48,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:18:48,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:48,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:52,757 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:18:52,759 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 36.66 2021-12-16 01:18:52,775 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:18:52,778 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:18:52,778 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:18:58,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:18:58,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:18:58,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:02,789 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.66 % 2021-12-16 01:19:02,789 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 36.66 2021-12-16 01:19:02,809 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:02,811 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:02,812 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:08,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.97 % 2021-12-16 01:19:08,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:08,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:12,825 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:12,825 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.22 2021-12-16 01:19:12,845 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:12,848 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:12,848 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:18,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:18,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:18,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:22,861 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:22,861 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.22 2021-12-16 01:19:22,878 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:22,881 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:22,881 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:28,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:28,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:28,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:32,894 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:32,894 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.22 2021-12-16 01:19:32,913 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:32,914 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:32,914 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:38,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:38,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:38,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:42,926 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:42,926 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.22 2021-12-16 01:19:42,943 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:42,944 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:42,944 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:48,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.22 % 2021-12-16 01:19:48,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:48,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:52,956 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.53 % 2021-12-16 01:19:52,956 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.53 2021-12-16 01:19:52,973 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:19:52,973 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:19:52,974 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:19:58,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.53 % 2021-12-16 01:19:58,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:19:58,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:03,003 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.53 % 2021-12-16 01:20:03,003 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.53 2021-12-16 01:20:03,022 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:03,025 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:03,026 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:08,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:08,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:08,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:13,039 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:13,039 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.79 2021-12-16 01:20:13,055 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:13,056 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:13,056 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:18,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:18,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:18,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:23,069 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:23,069 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.79 2021-12-16 01:20:23,088 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:23,092 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:23,092 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:28,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:28,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:28,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:33,102 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:33,104 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.79 2021-12-16 01:20:33,119 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:33,122 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:33,122 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:38,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:38,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:38,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:43,135 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:43,135 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.79 2021-12-16 01:20:43,154 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:43,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:43,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:48,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:48,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:48,645 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:53,168 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:53,168 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 37.79 2021-12-16 01:20:53,187 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:20:53,190 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:20:53,190 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:20:58,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.79 % 2021-12-16 01:20:58,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:20:58,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:03,201 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.10 % 2021-12-16 01:21:03,203 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 38.1 2021-12-16 01:21:03,217 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:03,220 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:03,220 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:08,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:08,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:08,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:13,232 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:13,233 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 38.36 2021-12-16 01:21:13,255 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:13,256 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:13,256 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:18,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:18,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:18,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:23,269 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:23,269 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 38.36 2021-12-16 01:21:23,286 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:23,289 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:23,289 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:28,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:28,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:28,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:33,299 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:33,299 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 38.36 2021-12-16 01:21:33,319 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:33,321 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:33,322 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:38,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:38,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:38,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:43,336 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.36 % 2021-12-16 01:21:43,336 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 38.36 2021-12-16 01:21:43,355 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:43,358 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:43,358 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:48,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.86 % 2021-12-16 01:21:48,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:48,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:53,372 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.42 % 2021-12-16 01:21:53,372 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.42 2021-12-16 01:21:53,392 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:21:53,397 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:21:53,397 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:21:58,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.42 % 2021-12-16 01:21:58,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:21:58,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:03,410 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.42 % 2021-12-16 01:22:03,410 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.42 2021-12-16 01:22:03,434 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:03,437 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:03,437 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:08,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:08,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:08,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:13,450 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:13,450 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.68 2021-12-16 01:22:13,464 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:13,467 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:13,467 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:18,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:18,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:18,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:23,480 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:23,480 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.68 2021-12-16 01:22:23,497 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:23,500 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:23,500 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:28,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:28,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:28,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:33,551 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:33,551 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.68 2021-12-16 01:22:33,566 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:33,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:33,569 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:38,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:38,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:38,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:43,582 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:43,582 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.68 2021-12-16 01:22:43,601 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:43,604 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:43,604 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:48,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:48,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:48,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:53,615 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.68 % 2021-12-16 01:22:53,615 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.68 2021-12-16 01:22:53,634 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:22:53,637 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:22:53,638 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:22:58,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-16 01:22:58,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:22:58,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:03,648 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-16 01:23:03,650 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.99 2021-12-16 01:23:03,665 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:03,667 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:03,667 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:08,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-16 01:23:08,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:08,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:13,678 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.99 % 2021-12-16 01:23:13,680 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 39.99 2021-12-16 01:23:13,697 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:13,698 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:13,700 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:18,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-16 01:23:18,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:18,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:23,713 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-16 01:23:23,713 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.25 2021-12-16 01:23:23,730 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:23,733 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:23,733 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:29,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-16 01:23:29,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:29,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:33,746 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.25 % 2021-12-16 01:23:33,746 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.25 2021-12-16 01:23:33,760 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:33,763 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:33,763 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:39,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:23:39,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:39,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:43,776 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:23:43,776 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.56 2021-12-16 01:23:43,798 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:43,801 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:43,802 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:49,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:23:49,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:49,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:53,812 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:23:53,812 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.56 2021-12-16 01:23:53,839 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:23:53,842 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:23:53,842 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:23:59,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:23:59,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:23:59,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:03,854 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:24:03,855 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.56 2021-12-16 01:24:03,872 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:03,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:03,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:09,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:24:09,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:09,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:13,888 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.56 % 2021-12-16 01:24:13,888 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.56 2021-12-16 01:24:13,904 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:13,908 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:13,908 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:19,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:19,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:19,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:23,921 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:23,921 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.81 2021-12-16 01:24:23,937 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:23,940 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:23,940 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:29,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:29,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:29,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:33,951 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:33,953 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.81 2021-12-16 01:24:33,969 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:33,970 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:33,970 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:39,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:39,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:39,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:43,983 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.81 % 2021-12-16 01:24:43,983 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 40.81 2021-12-16 01:24:43,997 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:44,000 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:44,000 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:49,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:24:49,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:49,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:54,016 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:24:54,016 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 41.12 2021-12-16 01:24:54,032 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:24:54,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:24:54,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:24:59,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:24:59,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:24:59,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:04,049 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:25:04,049 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 41.12 2021-12-16 01:25:04,068 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:04,071 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:04,071 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:09,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:25:09,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:09,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:14,084 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.12 % 2021-12-16 01:25:14,084 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 41.12 2021-12-16 01:25:14,108 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:14,111 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:14,111 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:19,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.31 % 2021-12-16 01:25:19,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:19,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:24,122 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.87 % 2021-12-16 01:25:24,124 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 41.87 2021-12-16 01:25:24,140 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:24,142 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:24,142 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:29,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.13 % 2021-12-16 01:25:29,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:29,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:34,154 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:34,155 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.44 2021-12-16 01:25:34,171 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:34,174 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:34,174 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:39,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:39,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:39,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:44,184 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:44,184 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.44 2021-12-16 01:25:44,201 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:44,204 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:44,204 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:49,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:49,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:49,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:54,216 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:54,216 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.44 2021-12-16 01:25:54,232 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:25:54,233 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:25:54,233 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:25:59,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:25:59,318 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:25:59,318 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:04,246 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.44 % 2021-12-16 01:26:04,246 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.44 2021-12-16 01:26:04,266 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:04,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:04,269 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:09,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:09,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:09,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:14,283 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:14,283 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.75 2021-12-16 01:26:14,299 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:14,302 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:14,302 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:19,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:19,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:19,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:24,313 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:24,315 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.75 2021-12-16 01:26:24,332 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:24,335 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:24,335 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:29,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:29,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:29,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:34,345 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 01:26:34,345 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 42.75 2021-12-16 01:26:34,364 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:34,365 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:34,367 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:39,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 01:26:39,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:39,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:44,378 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 01:26:44,378 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.01 2021-12-16 01:26:44,398 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:44,401 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:44,401 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:49,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 01:26:49,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:49,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:54,411 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 01:26:54,411 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.01 2021-12-16 01:26:54,428 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:26:54,431 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:26:54,433 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:26:59,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.20 % 2021-12-16 01:26:59,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:26:59,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:04,443 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.20 % 2021-12-16 01:27:04,444 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.2 2021-12-16 01:27:04,463 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:04,464 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:04,464 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:09,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.51 % 2021-12-16 01:27:09,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:09,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:14,474 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.51 % 2021-12-16 01:27:14,476 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.51 2021-12-16 01:27:14,493 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:14,494 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:14,496 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:19,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.82 % 2021-12-16 01:27:19,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:19,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:24,506 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.82 % 2021-12-16 01:27:24,506 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.82 2021-12-16 01:27:24,526 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:24,529 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:24,529 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:29,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.82 % 2021-12-16 01:27:29,499 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:29,499 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:34,540 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.82 % 2021-12-16 01:27:34,540 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 43.82 2021-12-16 01:27:34,558 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:34,561 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:34,561 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:39,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.82 % 2021-12-16 01:27:39,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:39,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:44,573 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:27:44,573 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:27:44,595 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:44,598 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:44,598 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:49,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:27:49,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:49,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:54,609 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:27:54,609 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:27:54,625 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:27:54,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:27:54,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:27:59,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:27:59,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:27:59,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:04,640 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:04,640 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:28:04,657 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:04,657 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:04,658 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:09,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:09,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:09,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:14,671 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:14,671 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:28:14,690 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:14,691 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:14,691 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:19,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:19,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:19,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:24,703 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:24,703 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:28:24,726 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:24,729 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:24,729 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:29,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:29,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:29,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:34,742 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:34,742 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.08 2021-12-16 01:28:34,759 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:34,760 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:34,760 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:39,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.08 % 2021-12-16 01:28:39,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:39,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:44,773 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.34 % 2021-12-16 01:28:44,773 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.34 2021-12-16 01:28:44,792 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:44,795 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:44,795 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:49,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.34 % 2021-12-16 01:28:49,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:49,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:54,821 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.34 % 2021-12-16 01:28:54,821 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.34 2021-12-16 01:28:54,838 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:28:54,842 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:28:54,842 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:28:59,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.65 % 2021-12-16 01:28:59,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:28:59,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:04,854 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.65 % 2021-12-16 01:29:04,855 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.65 2021-12-16 01:29:04,872 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:04,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:04,875 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:09,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.65 % 2021-12-16 01:29:09,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:09,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:14,884 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.65 % 2021-12-16 01:29:14,884 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.65 2021-12-16 01:29:14,901 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:14,904 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:14,904 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:19,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.65 % 2021-12-16 01:29:19,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:19,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:24,915 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-16 01:29:24,915 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.96 2021-12-16 01:29:24,931 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:24,934 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:24,934 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:29,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-16 01:29:29,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:29,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:34,950 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-16 01:29:34,951 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.96 2021-12-16 01:29:34,977 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:34,983 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:34,983 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:39,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-16 01:29:39,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:39,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:44,996 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.96 % 2021-12-16 01:29:44,996 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 44.96 2021-12-16 01:29:45,013 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:45,016 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:45,016 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:49,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:29:49,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:49,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:55,028 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:29:55,029 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.21 2021-12-16 01:29:55,052 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:29:55,055 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:29:55,055 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:29:59,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:29:59,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:29:59,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:05,078 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:30:05,079 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.21 2021-12-16 01:30:05,095 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:05,098 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:05,098 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:09,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:30:09,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:09,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:15,108 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.21 % 2021-12-16 01:30:15,108 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.21 2021-12-16 01:30:15,127 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:15,128 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:15,128 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:19,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.59 % 2021-12-16 01:30:19,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:19,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:25,140 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.78 % 2021-12-16 01:30:25,141 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.78 2021-12-16 01:30:25,155 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:25,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:25,157 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:29,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.96 % 2021-12-16 01:30:29,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:29,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:35,170 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.96 % 2021-12-16 01:30:35,171 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.96 2021-12-16 01:30:35,187 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:35,191 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:35,191 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:39,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.96 % 2021-12-16 01:30:39,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:39,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:45,203 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.96 % 2021-12-16 01:30:45,203 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 45.96 2021-12-16 01:30:45,220 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:45,221 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:45,223 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:49,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.27 % 2021-12-16 01:30:49,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:49,872 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:55,234 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.53 % 2021-12-16 01:30:55,234 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.53 2021-12-16 01:30:55,256 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:30:55,259 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:30:55,259 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:30:59,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.53 % 2021-12-16 01:30:59,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:30:59,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:05,269 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.53 % 2021-12-16 01:31:05,269 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.53 2021-12-16 01:31:05,286 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:05,290 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:05,290 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:09,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.53 % 2021-12-16 01:31:09,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:31:09,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:15,302 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:15,302 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.84 2021-12-16 01:31:15,326 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:15,328 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:15,328 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:19,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:19,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:31:19,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:25,342 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:25,342 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.84 2021-12-16 01:31:25,361 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:25,365 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:25,365 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:29,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:29,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:31:29,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:35,377 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:35,377 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.84 2021-12-16 01:31:35,397 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:35,398 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:35,398 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:39,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:39,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:31:39,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:45,410 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:45,410 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.84 2021-12-16 01:31:45,440 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:45,443 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:45,444 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:49,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:50,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:31:50,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:31:55,454 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:31:55,454 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 46.84 2021-12-16 01:31:55,470 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:31:55,471 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:31:55,471 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:00,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.84 % 2021-12-16 01:32:00,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:00,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:05,484 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.15 % 2021-12-16 01:32:05,484 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.15 2021-12-16 01:32:05,510 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:05,515 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:05,515 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:10,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.15 % 2021-12-16 01:32:10,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:10,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:15,525 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.15 % 2021-12-16 01:32:15,526 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.15 2021-12-16 01:32:15,542 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:15,545 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:15,545 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:20,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.15 % 2021-12-16 01:32:20,056 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:20,056 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:25,555 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:25,555 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:32:25,573 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:25,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:25,575 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:30,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:30,081 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:30,081 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:35,591 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:35,592 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:32:35,612 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:35,615 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:35,615 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:40,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:40,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:40,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:45,627 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:45,627 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:32:45,641 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:45,644 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:45,644 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:50,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:50,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:32:50,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:32:55,668 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:32:55,668 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:32:55,686 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:32:55,688 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:32:55,688 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:00,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:00,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:00,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:05,700 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:05,700 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:33:05,714 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:05,717 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:05,717 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:10,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:10,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:10,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:15,727 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:15,727 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:33:15,746 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:15,749 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:15,749 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:20,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:20,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:20,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:25,759 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.41 % 2021-12-16 01:33:25,759 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 47.41 2021-12-16 01:33:25,778 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:25,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:25,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:30,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.04 % 2021-12-16 01:33:30,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:30,191 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:35,793 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-16 01:33:35,793 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.23 2021-12-16 01:33:35,812 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:35,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:35,815 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:40,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-16 01:33:40,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:40,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:45,826 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:33:45,826 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:33:45,845 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:45,848 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:45,848 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:50,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:33:50,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:33:50,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:33:55,859 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:33:55,859 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:33:55,875 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:33:55,878 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:33:55,878 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:00,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:00,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:00,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:05,890 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:05,890 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:34:05,907 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:05,910 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:05,910 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:10,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:10,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:10,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:15,921 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:15,921 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:34:15,937 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:15,938 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:15,938 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:20,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:20,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:20,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:25,950 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:25,950 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:34:25,973 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:25,974 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:25,974 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:30,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:30,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:30,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:35,984 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 01:34:35,986 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.42 2021-12-16 01:34:36,003 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:36,006 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:36,006 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:40,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.68 % 2021-12-16 01:34:40,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:40,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:46,016 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.68 % 2021-12-16 01:34:46,016 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.68 2021-12-16 01:34:46,035 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:46,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:46,036 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:50,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.68 % 2021-12-16 01:34:50,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:34:50,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:34:56,048 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.68 % 2021-12-16 01:34:56,048 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.68 2021-12-16 01:34:56,068 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:34:56,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:34:56,072 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:00,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:00,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:00,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:06,084 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:06,085 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:06,102 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:06,105 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:06,105 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:10,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:10,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:10,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:16,122 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:16,122 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:16,150 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:16,154 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:16,154 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:20,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:20,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:20,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:26,165 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:26,165 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:26,184 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:26,188 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:26,188 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:30,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:30,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:30,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:36,201 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:36,201 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:36,226 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:36,227 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:36,227 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:40,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:40,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:40,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:46,242 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:46,242 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:46,259 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:46,263 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:46,263 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:50,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:50,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:35:50,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:35:56,273 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:35:56,273 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 48.93 2021-12-16 01:35:56,299 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:35:56,301 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:35:56,301 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:00,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.93 % 2021-12-16 01:36:00,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:00,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:06,311 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:06,311 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.19 2021-12-16 01:36:06,326 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:06,329 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:06,329 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:10,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:10,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:10,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:16,341 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:16,341 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.19 2021-12-16 01:36:16,357 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:16,358 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:16,358 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:20,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:20,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:20,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:26,369 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:26,369 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.19 2021-12-16 01:36:26,387 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:26,390 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:26,390 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:30,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:30,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:30,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:36,401 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:36,401 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.19 2021-12-16 01:36:36,417 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:36,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:36,421 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:40,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:40,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:40,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:46,431 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.19 % 2021-12-16 01:36:46,433 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.19 2021-12-16 01:36:46,453 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:46,454 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:46,456 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:50,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.38 % 2021-12-16 01:36:50,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:36:50,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:36:56,466 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.57 % 2021-12-16 01:36:56,466 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.57 2021-12-16 01:36:56,482 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:36:56,484 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:36:56,484 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:00,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.75 % 2021-12-16 01:37:00,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:00,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:06,496 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.94 % 2021-12-16 01:37:06,497 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 49.94 2021-12-16 01:37:06,515 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:06,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:06,516 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:10,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.94 % 2021-12-16 01:37:10,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:10,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:16,528 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.20 % 2021-12-16 01:37:16,528 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.2 2021-12-16 01:37:16,546 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:16,549 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:16,549 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:20,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.20 % 2021-12-16 01:37:20,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:20,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:26,562 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.20 % 2021-12-16 01:37:26,562 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.2 2021-12-16 01:37:26,579 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:26,582 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:26,584 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:30,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.20 % 2021-12-16 01:37:30,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:30,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:36,595 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.20 % 2021-12-16 01:37:36,595 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.2 2021-12-16 01:37:36,615 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:36,615 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:36,617 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:40,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:37:40,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:40,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:46,630 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:37:46,630 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:37:46,651 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:46,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:46,654 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:50,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:37:50,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:37:50,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:37:56,664 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:37:56,665 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:37:56,684 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:37:56,686 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:37:56,686 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:00,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:00,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:00,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:06,697 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:06,697 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:38:06,713 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:06,716 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:06,716 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:10,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:10,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:10,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:16,726 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:16,726 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:38:16,743 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:16,746 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:16,746 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:20,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:20,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:20,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:26,756 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:26,757 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:38:26,779 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:26,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:26,780 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:30,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:30,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:30,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:36,792 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.46 % 2021-12-16 01:38:36,792 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.46 2021-12-16 01:38:36,823 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:36,826 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:36,826 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:40,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:38:40,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:40,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:46,839 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:38:46,839 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:38:46,861 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:46,865 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:46,865 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:50,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:38:50,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:38:50,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:38:56,885 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:38:56,885 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:38:56,904 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:38:56,907 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:38:56,907 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:00,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:00,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:00,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:06,921 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:06,921 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:39:06,944 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:06,947 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:06,947 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:10,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:10,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:10,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:16,959 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:16,959 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:39:16,979 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:16,983 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:16,983 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:20,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:20,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:20,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:26,996 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:26,996 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:39:27,016 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:27,020 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:27,020 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:30,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:30,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:30,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:37,032 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:37,032 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:39:37,049 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:37,052 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:37,052 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:40,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:40,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:40,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:47,062 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.72 % 2021-12-16 01:39:47,062 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.72 2021-12-16 01:39:47,079 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:47,082 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:47,082 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:50,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.97 % 2021-12-16 01:39:50,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:39:50,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:39:57,092 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.97 % 2021-12-16 01:39:57,092 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 50.97 2021-12-16 01:39:57,108 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:39:57,109 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:39:57,109 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:00,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.97 % 2021-12-16 01:40:00,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:00,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:07,119 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.35 % 2021-12-16 01:40:07,119 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 51.35 2021-12-16 01:40:07,135 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:07,138 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:07,138 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:10,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.54 % 2021-12-16 01:40:10,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:10,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:17,148 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.98 % 2021-12-16 01:40:17,148 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 51.98 2021-12-16 01:40:17,164 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:17,165 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:17,165 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:20,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.98 % 2021-12-16 01:40:20,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:20,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:27,177 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.98 % 2021-12-16 01:40:27,177 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 51.98 2021-12-16 01:40:27,207 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:27,210 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:27,210 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:31,005 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.98 % 2021-12-16 01:40:31,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:31,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:37,220 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:40:37,220 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.27 2021-12-16 01:40:37,240 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:37,243 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:37,243 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:41,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:40:41,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:41,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:47,259 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:40:47,259 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.27 2021-12-16 01:40:47,276 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:47,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:47,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:51,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:40:51,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:40:51,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:40:57,289 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:40:57,290 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.27 2021-12-16 01:40:57,306 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:40:57,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:40:57,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:01,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:41:01,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:01,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:07,319 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:41:07,321 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.27 2021-12-16 01:41:07,338 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:07,341 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:07,342 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:11,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:41:11,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:11,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:17,352 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:41:17,354 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.27 2021-12-16 01:41:17,377 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:17,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:17,378 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:21,117 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.27 % 2021-12-16 01:41:21,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:21,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:27,390 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:27,390 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:41:27,411 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:27,413 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:27,413 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:31,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:31,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:31,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:37,427 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:37,427 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:41:37,444 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:37,448 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:37,448 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:41,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:41,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:41,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:47,460 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:47,460 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:41:47,482 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:47,483 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:47,483 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:51,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:51,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:41:51,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:41:57,496 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:41:57,496 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:41:57,516 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:41:57,519 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:41:57,519 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:01,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:01,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:01,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:07,529 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:07,529 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:07,549 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:07,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:07,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:11,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:11,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:11,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:17,565 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:17,565 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:17,588 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:17,591 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:17,591 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:21,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:21,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:21,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:27,602 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:27,602 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:27,622 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:27,625 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:27,625 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:31,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:31,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:31,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:37,638 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:37,638 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:37,654 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:37,657 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:37,657 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:41,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:41,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:41,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:47,668 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:47,670 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:47,694 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:47,697 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:47,697 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:51,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:51,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:42:51,283 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:42:57,710 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:42:57,710 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:42:57,729 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:42:57,730 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:42:57,730 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:01,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:43:01,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:01,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:07,744 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:43:07,744 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 52.52 2021-12-16 01:43:07,769 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:07,772 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:07,772 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:11,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.52 % 2021-12-16 01:43:11,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:11,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:17,785 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.16 % 2021-12-16 01:43:17,785 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 53.16 2021-12-16 01:43:17,822 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:17,826 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:17,828 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:21,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-16 01:43:21,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:21,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:27,838 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.35 % 2021-12-16 01:43:27,838 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 53.35 2021-12-16 01:43:27,857 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:27,859 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:27,859 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:31,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.53 % 2021-12-16 01:43:31,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:31,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:37,869 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.79 % 2021-12-16 01:43:37,869 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 53.79 2021-12-16 01:43:37,884 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:37,887 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:37,887 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:41,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.79 % 2021-12-16 01:43:41,374 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:41,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:47,898 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.79 % 2021-12-16 01:43:47,898 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 53.79 2021-12-16 01:43:47,918 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:47,921 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:47,921 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:51,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.79 % 2021-12-16 01:43:51,390 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:43:51,390 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:43:57,936 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:43:57,936 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:43:57,951 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:43:57,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:43:57,953 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:01,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:01,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:01,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:07,970 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:07,970 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:44:07,990 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:07,992 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:07,992 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:11,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:11,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:11,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:18,002 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:18,003 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:44:18,019 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:18,020 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:18,020 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:21,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:21,443 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:21,443 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:28,032 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:28,032 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:44:28,056 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:28,059 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:28,059 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:31,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:31,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:31,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:38,071 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:38,071 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:44:38,088 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:38,091 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:38,092 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:41,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:41,484 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:41,484 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:48,102 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.05 % 2021-12-16 01:44:48,102 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.05 2021-12-16 01:44:48,128 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:48,130 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:48,131 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:51,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:44:51,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:44:51,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:44:58,141 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:44:58,141 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:44:58,174 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:44:58,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:44:58,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:01,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:01,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:01,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:08,187 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:08,187 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:08,204 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:08,207 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:08,207 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:11,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:11,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:11,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:18,219 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:18,219 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:18,234 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:18,236 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:18,236 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:21,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:21,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:21,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:28,247 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:28,247 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:28,276 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:28,282 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:28,282 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:31,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:31,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:31,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:38,295 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:38,295 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:38,309 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:38,312 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:38,312 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:41,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:41,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:41,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:48,323 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:48,323 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:48,342 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:48,344 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:48,345 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:51,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:51,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:45:51,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:45:58,355 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:45:58,355 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:45:58,372 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:45:58,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:45:58,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:01,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:46:01,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:01,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:08,388 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:46:08,388 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:46:08,404 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:08,405 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:08,405 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:11,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:46:11,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:11,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:18,423 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:46:18,423 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.31 2021-12-16 01:46:18,444 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:18,450 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:18,450 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:21,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.31 % 2021-12-16 01:46:21,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:21,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:28,463 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.68 % 2021-12-16 01:46:28,463 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 54.68 2021-12-16 01:46:28,480 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:28,483 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:28,483 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:31,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 01:46:31,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:31,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:38,496 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.31 % 2021-12-16 01:46:38,496 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.31 2021-12-16 01:46:38,510 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:38,512 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:38,513 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:41,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.31 % 2021-12-16 01:46:41,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:41,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:48,526 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.31 % 2021-12-16 01:46:48,526 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.31 2021-12-16 01:46:48,546 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:48,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:48,552 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:51,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.31 % 2021-12-16 01:46:51,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:46:51,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:46:58,565 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-16 01:46:58,565 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-16 01:46:58,582 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:46:58,584 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:46:58,584 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:01,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-16 01:47:01,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:01,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:08,595 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-16 01:47:08,596 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-16 01:47:08,619 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:08,622 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:08,622 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:11,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-16 01:47:11,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:11,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:18,635 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.57 % 2021-12-16 01:47:18,637 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.57 2021-12-16 01:47:18,657 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:18,661 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:18,661 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:21,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:21,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:21,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:28,673 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:28,674 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.83 2021-12-16 01:47:28,690 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:28,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:28,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:31,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:31,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:31,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:38,709 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:38,709 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.83 2021-12-16 01:47:38,727 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:38,730 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:38,730 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:41,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:41,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:41,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:48,746 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:48,746 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.83 2021-12-16 01:47:48,763 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:48,765 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:48,766 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:51,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:51,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:47:51,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:47:58,776 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:47:58,778 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.83 2021-12-16 01:47:58,796 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:47:58,798 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:47:58,798 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:01,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:48:01,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:01,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:08,808 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:48:08,809 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 55.83 2021-12-16 01:48:08,825 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:08,828 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:08,828 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:11,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.83 % 2021-12-16 01:48:11,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:11,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:18,839 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:18,839 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:48:18,855 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:18,857 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:18,857 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:21,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:21,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:21,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:28,888 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:28,888 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:48:28,907 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:28,911 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:28,911 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:31,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:31,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:31,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:38,924 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:38,924 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:48:38,940 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:38,944 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:38,944 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:41,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:41,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:41,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:48,953 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:48,953 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:48:48,970 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:48,973 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:48,973 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:51,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:51,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:48:51,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:48:58,986 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:48:58,986 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:48:59,019 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:48:59,025 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:48:59,025 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:01,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:01,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:01,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:09,039 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:09,039 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:49:09,062 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:09,065 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:09,065 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:11,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:11,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:11,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:19,078 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:19,078 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:49:19,098 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:19,099 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:19,099 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:22,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:22,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:22,028 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:29,112 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:29,112 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:49:29,132 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:29,134 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:29,134 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:32,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:32,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:32,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:39,148 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:39,148 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:49:39,173 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:39,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:39,177 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:42,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:42,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:42,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:49,190 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:49,191 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.09 2021-12-16 01:49:49,213 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:49,216 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:49,217 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:52,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.09 % 2021-12-16 01:49:52,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:49:52,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:49:59,229 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.72 % 2021-12-16 01:49:59,229 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 56.72 2021-12-16 01:49:59,246 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:49:59,247 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:49:59,247 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:02,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.72 % 2021-12-16 01:50:02,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:02,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:09,259 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.10 % 2021-12-16 01:50:09,259 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.1 2021-12-16 01:50:09,276 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:09,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:09,279 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:12,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.10 % 2021-12-16 01:50:12,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:12,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:19,289 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.36 % 2021-12-16 01:50:19,289 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.36 2021-12-16 01:50:19,305 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:19,306 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:19,308 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:22,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.36 % 2021-12-16 01:50:22,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:22,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:29,319 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.61 % 2021-12-16 01:50:29,319 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.61 2021-12-16 01:50:29,336 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:29,342 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:29,342 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:32,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.61 % 2021-12-16 01:50:32,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:32,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:39,365 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.61 % 2021-12-16 01:50:39,365 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.61 2021-12-16 01:50:39,385 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:39,388 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:39,388 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:42,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.61 % 2021-12-16 01:50:42,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:42,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:49,401 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:50:49,401 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:50:49,421 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:49,424 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:49,424 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:52,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:50:52,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:50:52,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:50:59,436 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:50:59,436 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:50:59,453 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:50:59,456 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:50:59,456 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:02,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:02,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:02,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:09,467 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:09,467 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:09,486 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:09,489 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:09,490 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:12,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:12,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:12,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:19,506 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:19,506 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:19,523 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:19,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:19,526 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:22,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:22,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:22,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:29,538 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:29,538 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:29,558 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:29,559 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:29,559 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:32,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:32,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:32,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:39,572 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:39,572 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:39,588 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:39,592 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:39,592 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:42,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:42,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:42,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:49,604 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:49,604 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:49,625 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:49,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:49,628 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:52,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:52,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:51:52,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:51:59,638 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:51:59,638 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:51:59,654 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:51:59,657 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:51:59,657 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:02,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:02,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:02,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:09,667 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:09,667 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:09,684 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:09,687 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:09,687 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:12,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:12,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:12,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:19,700 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:19,700 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:19,719 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:19,721 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:19,721 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:22,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:22,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:22,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:29,733 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:29,733 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:29,759 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:29,763 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:29,763 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:32,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:32,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:32,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:39,773 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:39,773 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:39,790 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:39,792 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:39,792 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:42,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:42,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:42,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:49,803 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:49,803 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:49,821 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:49,822 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:49,822 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:52,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:52,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:52:52,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:52:59,835 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:52:59,835 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:52:59,855 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:52:59,858 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:52:59,858 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:02,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:53:02,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:02,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:09,871 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.87 % 2021-12-16 01:53:09,871 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 57.87 2021-12-16 01:53:09,887 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:53:09,890 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:53:09,890 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:12,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.18 % 2021-12-16 01:53:12,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:12,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:19,901 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.56 % 2021-12-16 01:53:19,903 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.56 2021-12-16 01:53:19,924 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:53:19,927 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:53:19,927 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:22,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.56 % 2021-12-16 01:53:22,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:22,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:29,941 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:29,943 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:53:29,967 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:53:29,969 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:53:29,969 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:32,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:32,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:32,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:39,980 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:39,982 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:53:40,003 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:53:40,006 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:53:40,006 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:42,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:42,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:42,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:50,019 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:50,019 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:53:50,043 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:53:50,046 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:53:50,046 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:53:52,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:53:52,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:53:52,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:00,058 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:00,058 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:00,073 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:00,075 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:00,075 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:02,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:02,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:02,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:10,088 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:10,088 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:10,105 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:10,108 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:10,108 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:12,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:12,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:12,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:20,121 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:20,121 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:20,144 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:20,145 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:20,145 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:22,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:22,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:22,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:30,158 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:30,158 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:30,177 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:30,180 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:30,180 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:32,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:32,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:32,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:40,191 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:40,191 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:40,210 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:40,213 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:40,213 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:42,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:42,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:42,645 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:50,230 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:50,230 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:54:50,253 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:54:50,256 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:54:50,256 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:54:52,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:54:52,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:54:52,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:00,266 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:00,266 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:00,296 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:00,299 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:00,299 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:02,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:02,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:02,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:10,312 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:10,312 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:10,328 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:10,329 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:10,329 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:12,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:12,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:12,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:20,341 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:20,341 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:20,359 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:20,362 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:20,362 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:22,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:22,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:22,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:30,372 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:30,374 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:30,388 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:30,391 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:30,391 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:32,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:32,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:32,737 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:40,404 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:40,404 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:40,427 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:40,430 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:40,430 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:42,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:42,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:42,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:50,441 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:50,441 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:55:50,459 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:55:50,463 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:55:50,463 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:55:52,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:55:52,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:55:52,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:00,473 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:56:00,473 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:56:00,489 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:00,492 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:00,492 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:02,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:56:02,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:02,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:10,503 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:56:10,503 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 58.74 2021-12-16 01:56:10,529 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:10,532 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:10,532 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:12,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.74 % 2021-12-16 01:56:12,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:12,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:20,543 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.06 % 2021-12-16 01:56:20,543 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.06 2021-12-16 01:56:20,559 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:20,562 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:20,562 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:22,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.37 % 2021-12-16 01:56:22,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:22,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:30,573 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.55 % 2021-12-16 01:56:30,573 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.55 2021-12-16 01:56:30,591 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:30,592 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:30,592 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:32,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:56:32,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:32,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:40,605 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:56:40,605 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:56:40,630 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:40,631 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:40,631 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:42,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:56:42,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:42,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:50,642 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:56:50,642 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:56:50,661 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:56:50,664 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:56:50,664 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:56:52,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:56:52,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:56:52,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:00,674 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:00,674 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:00,690 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:00,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:00,693 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:02,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:02,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:02,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:10,703 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:10,703 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:10,719 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:10,720 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:10,720 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:12,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:12,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:12,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:20,730 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:20,732 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:20,747 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:20,750 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:20,750 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:22,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:22,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:22,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:30,760 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:30,762 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:30,779 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:30,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:30,782 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:32,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:32,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:32,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:40,796 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:40,798 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:40,813 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:40,816 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:40,816 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:42,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:43,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:43,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:50,828 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:50,828 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:57:50,844 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:57:50,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:57:50,845 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:57:53,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:57:53,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:57:53,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:00,859 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:00,859 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:00,881 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:00,884 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:00,884 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:03,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:03,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:03,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:10,895 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:10,895 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:10,913 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:10,914 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:10,915 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:13,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:13,058 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:13,058 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:20,927 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:20,927 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:20,947 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:20,950 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:20,950 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:23,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:23,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:23,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:30,961 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:30,963 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:30,983 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:30,987 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:30,987 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:33,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:33,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:33,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:41,000 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:41,000 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:41,016 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:41,019 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:41,019 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:43,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:43,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:43,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:51,030 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:51,030 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:58:51,053 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:58:51,055 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:58:51,055 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:58:53,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:58:53,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:58:53,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:01,068 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:01,069 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:59:01,086 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:01,089 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:01,089 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:03,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:03,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:59:03,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:11,101 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:11,101 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:59:11,118 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:11,119 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:11,119 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:13,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:13,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:59:13,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:21,135 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:21,135 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.74 2021-12-16 01:59:21,160 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:21,164 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:21,164 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:23,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.74 % 2021-12-16 01:59:23,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:59:23,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:31,180 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.93 % 2021-12-16 01:59:31,180 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 59.93 2021-12-16 01:59:31,197 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:31,200 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:31,200 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:33,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.93 % 2021-12-16 01:59:33,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:59:33,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:41,213 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.55 % 2021-12-16 01:59:41,213 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.55 2021-12-16 01:59:41,229 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:41,232 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:41,232 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:43,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 01:59:43,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 01:59:43,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:51,246 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 01:59:51,246 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 01:59:51,266 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 01:59:51,267 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 01:59:51,267 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 01:59:53,242 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >60.74% in 5130.45300007 seconds 2021-12-16 01:59:53,256 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-16 01:59:53,256 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-16 01:59:53,257 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 01:59:53,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 01:59:53,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 01:59:53,677 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 01:59:53,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 01:59:53,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 01:59:53,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 01:59:53,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 01:59:53,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 01:59:53,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 01:59:53,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 01:59:53,782 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 01:59:53,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 01:59:53,842 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['214202 memcached\n'], []) 2021-12-16 01:59:53,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 01:59:53,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 01:59:53,930 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['214202 memcached\n'], []) 2021-12-16 01:59:53,930 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 01:59:53,933 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 01:59:54,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 01:59:54,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 01:59:54,339 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 01:59:54,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 01:59:54,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 01:59:54,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 01:59:54,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 01:59:54,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 01:59:54,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 01:59:54,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 01:59:54,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 01:59:54,448 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 01:59:54,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 01:59:54,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:54,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:54,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:54,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:54,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 01:59:54,591 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 01:59:54,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 01:59:54,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:54,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 01:59:54,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 01:59:54,628 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 01:59:54,631 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 01:59:54,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 01:59:54,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 01:59:55,033 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 01:59:55,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 01:59:55,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 01:59:55,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 01:59:55,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 01:59:55,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 01:59:55,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 01:59:55,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 01:59:55,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 01:59:55,170 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 01:59:55,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 01:59:55,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:55,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:55,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 01:59:55,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 01:59:55,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 01:59:55,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 01:59:55,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 01:59:55,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 01:59:55,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 01:59:55,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 01:59:55,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:55,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:55,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 01:59:55,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 01:59:56,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 01:59:56,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 01:59:56,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 01:59:56,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 01:59:56,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 01:59:56,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 01:59:56,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 01:59:56,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 01:59:56,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 01:59:56,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 01:59:56,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 01:59:56,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 01:59:56,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 01:59:56,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 01:59:56,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 01:59:56,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 01:59:56,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 01:59:56,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 01:59:56,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:56,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:56,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 01:59:56,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 01:59:56,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:57,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:57,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 01:59:57,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 01:59:57,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 01:59:57,078 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 01:59:57,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 01:59:57,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:57,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 01:59:57,124 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 01:59:57,125 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 01:59:57,127 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 01:59:57,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 01:59:57,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 01:59:57,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 01:59:57,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 01:59:57,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 01:59:57,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 01:59:57,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 01:59:57,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 01:59:57,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 01:59:57,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 01:59:57,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 01:59:57,786 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 01:59:57,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 01:59:57,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:57,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:57,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:57,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 01:59:57,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 01:59:58,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 01:59:58,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 01:59:58,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 01:59:58,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 01:59:58,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 01:59:58,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 01:59:58,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 01:59:58,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 01:59:58,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 01:59:58,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 01:59:58,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 01:59:58,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 01:59:58,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 01:59:58,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:58,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:58,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 01:59:58,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 01:59:58,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 01:59:59,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 01:59:59,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 01:59:59,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 01:59:59,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 01:59:59,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 01:59:59,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 01:59:59,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 01:59:59,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 01:59:59,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 01:59:59,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 01:59:59,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 01:59:59,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 01:59:59,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 01:59:59,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 01:59:59,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 01:59:59,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:00,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:00,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:00:00,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:00:00,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:00:00,101 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 02:00:00,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:00:00,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:00,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:00:00,164 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 02:00:00,164 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 02:00:00,168 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 02:00:00,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 02:00:00,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 02:00:00,660 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 02:00:00,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 02:00:00,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 02:00:00,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 02:00:00,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 02:00:00,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 02:00:00,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 02:00:00,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 02:00:00,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 02:00:00,813 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 02:00:00,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 02:00:00,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:00,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:00,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:00:00,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:00:00,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:01,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:00:01,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:00:01,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:01,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:00:01,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:00:01,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:01,280 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:01,280 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:01,306 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:01,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:01,309 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:00:01,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:00:01,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:00:01,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:01,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:00:01,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:00:01,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:00:01,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:00:01,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:00:01,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:00:01,628 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 02:00:01,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:00:01,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:00:01,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:00:01,684 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 02:00:01,684 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 02:00:01,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-16 02:00:01,691 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-16 02:00:11,322 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:11,322 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:11,338 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:11,339 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:11,339 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:00:21,352 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:21,354 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:21,372 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:21,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:21,375 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:00:31,387 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:31,387 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:31,410 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:31,411 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:31,411 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:00:41,421 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:41,423 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:41,444 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:41,446 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:41,446 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:00:51,457 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:00:51,457 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:00:51,476 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:00:51,477 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:00:51,477 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:01,490 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:01,490 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:01,505 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:01,507 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:01,507 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:01,701 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-16 02:01:01,701 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 02:01:01,710 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-16 02:01:01,720 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-16 02:01:01,721 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 02:01:01,730 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:01,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:01,737 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:03,749 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:03,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:03,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:05,785 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:05,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:05,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:07,805 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:07,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:07,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:09,822 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:09,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:09,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:11,519 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:11,519 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:11,540 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:11,542 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:11,542 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:11,842 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:11,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:11,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:13,859 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:13,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:13,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:15,878 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:15,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:15,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:17,898 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:17,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:17,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:19,917 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:19,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:19,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:21,555 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:21,555 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:21,578 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:21,579 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:21,581 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:21,937 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:21,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:21,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:23,957 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:23,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:23,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:25,974 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:25,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:25,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:27,993 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:28,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:28,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:30,016 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:30,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:30,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:31,591 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:31,591 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:31,608 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:31,611 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:31,611 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:32,038 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:32,043 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:32,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:34,055 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:34,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:34,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:36,075 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:36,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:36,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:38,095 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:38,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:38,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:40,117 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:40,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:40,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:41,624 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:41,624 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:41,644 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:41,645 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:41,645 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:42,135 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:42,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:42,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:44,154 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:44,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:44,163 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:46,174 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:46,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:46,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:48,194 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:48,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:48,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:50,213 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:50,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:50,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:51,657 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:01:51,657 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:01:51,680 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:01:51,681 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:01:51,681 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:01:52,234 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:52,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:52,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:54,255 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:54,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:54,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:56,273 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:56,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:56,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:01:58,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:01:58,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:01:58,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:00,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:00,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:00,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:01,694 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:02:01,694 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:02:01,711 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:02:01,713 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:02:01,713 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:02:02,328 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:02,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:02,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:04,348 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:04,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:04,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:06,369 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:06,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:06,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:08,391 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:08,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:08,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:10,408 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:10,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:10,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:11,724 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:02:11,724 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:02:11,740 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:02:11,742 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:02:11,742 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:02:12,430 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:12,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:12,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:14,451 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:14,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:14,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:16,469 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:16,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:16,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:18,490 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:18,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:18,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:25,497 | test | ERROR | MainThread | [rest_client:_http_request:832] Socket error while connecting to http://172.23.110.64:8091/nodes/self error timed out 2021-12-16 02:02:25,499 | test | ERROR | MainThread | [rest_client:is_ns_server_running:79] Server 172.23.110.64:8091 is unavailable 2021-12-16 02:02:25,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:25,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:30,180 | test | DEBUG | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.74 % 2021-12-16 02:02:30,180 | test | INFO | pool-3-thread-2 | [task:check:385] Rebalance - status: running, progress: 60.74 2021-12-16 02:02:30,184 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:30,191 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:30,191 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:30,204 | infra | DEBUG | pool-3-thread-2 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:02:30,207 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:02:30,207 | infra | DEBUG | pool-3-thread-2 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:02:32,203 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:32,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:32,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:34,223 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:34,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:34,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:36,243 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:36,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:36,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:38,260 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:38,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:38,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:40,217 | test | ERROR | pool-3-thread-2 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'91173dc90dd28bf9a4d1521b930c6267', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=f8285855f318fdf046bfad774af3bb44', u'status': u'notRunning'} - rebalance failed 2021-12-16 02:02:40,250 | test | INFO | pool-3-thread-2 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.67: 2021-12-16 02:02:40,250 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639648950170L, u'shortText': u'message', u'serverTime': u'2021-12-16T02:02:30.170Z', u'text': u'Rebalance exited with reason {{badmatch,\n {leader_activities_error,\n {default,rebalance},\n {quorum_lost,\n {lease_lost,\'ns_1@172.23.110.68\'}}}},\n [{ns_rebalancer,rebalance,5,\n [{file,"src/ns_rebalancer.erl"},{line,450}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]}.\nRebalance Operation Id = 64ac5053fa7988717e61030da5df4eba'} 2021-12-16 02:02:40,250 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'mb_master', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1639648948895L, u'shortText': u'message', u'serverTime': u'2021-12-16T02:02:28.895Z', u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over."} 2021-12-16 02:02:40,250 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639648807273L, u'shortText': u'message', u'serverTime': u'2021-12-16T02:00:07.273Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1216 00:29:28.847954 214435 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 02:02:40,250 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639648807256L, u'shortText': u'message', u'serverTime': u'2021-12-16T02:00:07.256Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-16 02:02:40,250 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643543232L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:32:23.232Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-16 02:02:40,252 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643543056L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:32:23.056Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-16 02:02:40,252 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643542800L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:32:22.800Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.68','ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.66'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 64ac5053fa7988717e61030da5df4eba"} 2021-12-16 02:02:40,252 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643530585L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:32:10.585Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 138 seconds.'} 2021-12-16 02:02:40,252 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'mb_master', u'type': u'info', u'node': u'ns_1@172.23.110.65', u'tstamp': 1639643500253L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:31:40.253Z', u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over."} 2021-12-16 02:02:40,252 | test | ERROR | pool-3-thread-2 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639643360347L, u'shortText': u'message', u'serverTime': u'2021-12-16T00:29:20.347Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.29391.174>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,504,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.68:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.29109.174>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.65\',\n \'ns_1@172.23.110.68\'],\n 86}},\n infinity]}}}}}.\nRebalance Operation Id = 787abf80acfd0a5e0fcb8a2c765814bf'} 2021-12-16 02:02:40,253 | test | ERROR | pool-3-thread-2 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'91173dc90dd28bf9a4d1521b930c6267', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=f8285855f318fdf046bfad774af3bb44', u'status': u'notRunning'} - rebalance failed 2021-12-16 02:02:40,279 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:40,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:40,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:42,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:42,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:42,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:44,318 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:44,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:44,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:47,726 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:47,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:47,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:49,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:49,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:49,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:51,766 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:51,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:51,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:53,786 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:53,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:53,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:55,802 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:55,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:55,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:57,822 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:57,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:57,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:02:59,844 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:02:59,849 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:02:59,849 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:03:01,864 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:03:01,869 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:03:01,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:03:03,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:03:03,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:03:03,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:03:05,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 02:03:05,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 02:03:05,910 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 02:03:07,921 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-16 02:03:07,930 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-16 02:03:07,930 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 02:03:07,940 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-16 02:03:07,950 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.68:8091 2021-12-16 02:03:07,950 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 02:03:07,960 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.68:8091 is running 2021-12-16 02:03:07,961 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 02:03:08,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 02:03:08,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 02:03:08,398 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 02:03:08,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 02:03:08,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 02:03:08,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 02:03:08,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 02:03:08,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 02:03:08,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 02:03:08,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 02:03:08,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 02:03:08,513 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 02:03:08,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 02:03:08,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:08,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:08,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:08,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:08,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:03:08,711 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 02:03:08,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:03:08,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:08,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:03:08,766 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 02:03:08,766 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 02:03:08,769 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 02:03:09,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 02:03:09,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 02:03:09,280 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 02:03:09,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 02:03:09,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 02:03:09,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 02:03:09,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 02:03:09,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 02:03:09,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 02:03:09,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 02:03:09,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 02:03:09,421 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 02:03:09,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 02:03:09,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:09,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:09,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:09,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:09,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:09,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:09,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:09,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:09,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:09,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:09,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:10,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:10,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:10,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:10,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:10,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:10,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:10,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:10,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 02:03:10,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 02:03:10,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 02:03:10,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 02:03:10,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 02:03:10,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 02:03:10,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 02:03:10,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 02:03:10,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:10,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:10,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 02:03:10,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 02:03:11,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 02:03:11,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 02:03:11,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 02:03:11,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 02:03:11,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 02:03:11,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 02:03:11,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 02:03:11,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 02:03:11,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 02:03:11,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 02:03:11,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:03:11,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:03:11,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:11,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 02:03:11,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 02:03:11,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:03:11,673 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 02:03:11,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:03:11,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:11,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:03:11,720 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 02:03:11,720 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 02:03:11,724 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 02:03:11,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 02:03:12,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 02:03:12,170 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 02:03:12,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 02:03:12,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 02:03:12,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 02:03:12,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 02:03:12,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 02:03:12,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 02:03:12,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 02:03:12,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 02:03:12,309 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 02:03:12,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 02:03:12,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:12,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:12,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:12,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:12,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:12,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:12,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:12,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:12,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:12,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:12,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:12,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:12,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:12,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 02:03:12,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 02:03:12,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 02:03:13,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 02:03:13,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 02:03:13,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 02:03:13,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 02:03:13,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 02:03:13,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 02:03:13,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 02:03:13,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 02:03:13,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 02:03:13,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 02:03:13,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 02:03:13,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 02:03:13,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 02:03:13,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 02:03:13,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 02:03:13,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 02:03:13,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 02:03:13,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:13,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:13,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:03:13,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 02:03:14,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:03:14,017 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 02:03:14,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:03:14,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:03:14,072 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 02:03:14,073 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 02:03:14,076 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 02:03:14,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 02:03:14,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 02:03:14,522 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 02:03:14,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 02:03:14,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 02:03:14,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 02:03:14,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 02:03:14,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 02:03:14,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 02:03:14,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 02:03:14,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 02:03:14,631 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 02:03:14,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 02:03:14,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:14,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:14,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 02:03:14,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:14,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:14,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 02:03:14,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:14,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:14,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 02:03:14,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:14,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:14,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 02:03:14,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:14,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:15,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:15,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 02:03:15,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:15,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 02:03:15,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:15,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 02:03:15,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 02:03:15,161 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 02:03:15,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 02:03:15,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 02:03:15,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 02:03:15,220 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 02:03:15,220 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 02:03:15,220 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639643542.74 2021-12-16 02:03:15,220 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 60 2021-12-16 02:03:15,256 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639648995.22 2021-12-16 02:03:15,256 | infra | INFO | pool-3-thread-3 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639648995.22' started 2021-12-16 02:03:15,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-16 02:03:15,262 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-16 02:03:15,285 | test | DEBUG | pool-3-thread-3 | [task:start_rebalance:356] Removing node 172.23.110.66:8091 from cluster 2021-12-16 02:03:15,285 | test | DEBUG | pool-3-thread-3 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.66', 'user': 'Administrator'} 2021-12-16 02:03:15,342 | test | DEBUG | pool-3-thread-3 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 02:03:15,345 | test | INFO | pool-3-thread-3 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 6.95900327139 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 5.54119931276 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.653140633685 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 0.197307173922 | --- OUT ---> | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 0.407148018454 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-16 02:03:15,355 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:03:15,355 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:03:15,375 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:03:15,378 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:03:15,378 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:03:25,388 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:03:25,388 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:03:25,404 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:03:25,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:03:25,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:03:35,420 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:03:35,420 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:03:35,437 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:03:35,438 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:03:35,438 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:03:45,448 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:03:45,450 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:03:45,467 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:03:45,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:03:45,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:03:55,480 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:03:55,480 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:03:55,499 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:03:55,502 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:03:55,502 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:05,512 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:05,512 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:05,529 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:05,532 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:05,533 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:15,543 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:15,543 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:15,559 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:15,562 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:15,562 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:25,572 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:25,572 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:25,588 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:25,591 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:25,591 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:35,604 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:35,604 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:35,622 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:35,625 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:35,625 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:45,638 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:45,638 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:45,661 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:45,664 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:45,664 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:04:55,674 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:04:55,674 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:04:55,693 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:04:55,696 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:04:55,696 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:05,706 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:05,706 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:05,721 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:05,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:05,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:15,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:15,275 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 0.0 2021-12-16 02:05:15,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:15,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:05:15,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:15,736 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:15,736 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:15,752 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:15,753 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:15,753 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:25,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:25,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:05:25,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:25,766 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:25,766 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:25,789 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:25,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:25,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:35,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:35,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:05:35,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:35,802 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:35,803 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:35,819 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:35,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:35,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:45,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:45,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:05:45,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:45,832 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:45,834 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:45,851 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:45,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:45,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:55,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:55,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:05:55,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:05:55,864 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:05:55,865 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:05:55,882 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:05:55,884 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:05:55,884 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:05,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:06:05,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:05,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:05,898 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:06:05,898 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:06:05,923 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:05,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:05,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:15,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:06:15,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:15,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:15,938 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 02:06:15,938 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 02:06:15,956 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:15,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:15,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:25,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:25,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:25,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:25,971 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:25,971 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:06:25,994 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:25,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:25,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:35,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:35,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:35,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:36,009 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:36,009 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:06:36,032 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:36,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:36,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:45,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:45,492 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:45,492 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:46,049 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:46,049 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:06:46,068 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:46,069 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:46,069 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:55,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:55,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:06:55,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:06:56,082 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:06:56,082 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:06:56,107 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:06:56,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:06:56,109 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:05,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:05,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:05,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:06,125 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:06,125 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:06,144 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:06,148 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:06,148 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:15,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:15,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:15,561 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:16,163 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:16,163 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:16,187 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:16,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:16,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:25,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:25,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:25,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:26,204 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:26,204 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:26,250 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:26,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:26,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:35,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:35,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:35,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:36,266 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:36,266 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:36,282 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:36,283 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:36,283 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:45,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:45,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:45,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:46,296 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:46,296 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:46,319 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:46,323 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:46,323 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:55,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:55,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:07:55,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:07:56,336 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:07:56,336 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:07:56,355 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:07:56,357 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:07:56,358 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:05,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:05,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:05,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:06,375 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:06,375 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:06,401 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:06,404 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:06,404 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:15,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:15,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:15,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:16,415 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:16,415 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:16,434 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:16,438 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:16,438 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:25,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:25,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:25,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:26,451 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:26,451 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:26,467 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:26,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:26,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:35,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:35,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:35,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:36,482 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:36,482 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:36,499 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:36,502 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:36,502 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:45,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:45,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:45,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:46,516 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:46,516 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:46,532 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:46,536 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:46,536 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:55,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:55,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:08:55,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:08:56,549 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:08:56,549 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:08:56,569 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:08:56,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:08:56,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:05,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:05,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:05,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:06,586 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:06,586 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:09:06,608 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:06,611 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:06,611 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:15,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:15,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:15,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:16,624 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:16,625 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:09:16,641 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:16,644 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:16,644 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:25,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:25,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:25,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:26,657 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:26,657 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:09:26,678 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:26,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:26,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:35,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:35,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:35,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:36,696 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.52 % 2021-12-16 02:09:36,696 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 1.52 2021-12-16 02:09:36,714 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:36,717 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:36,717 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:45,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 02:09:45,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:45,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:46,727 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.66 % 2021-12-16 02:09:46,729 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 2.66 2021-12-16 02:09:46,749 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:46,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:46,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:55,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:09:55,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:09:55,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:09:56,765 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:09:56,765 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:09:56,786 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:09:56,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:09:56,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:05,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:05,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:05,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:06,801 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:06,801 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:06,819 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:06,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:06,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:15,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:15,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:15,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:16,832 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:16,834 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:16,848 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:16,851 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:16,851 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:25,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:25,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:25,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:26,862 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:26,862 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:26,882 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:26,884 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:26,885 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:35,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:35,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:35,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:36,901 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:36,901 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:36,921 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:36,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:36,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:45,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:45,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:45,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:46,937 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:46,937 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:46,956 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:46,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:46,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:55,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:56,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:10:56,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:10:56,970 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:10:56,970 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:10:56,989 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:10:56,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:10:56,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:06,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:06,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:06,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:07,006 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:07,006 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:07,029 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:07,030 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:07,030 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:16,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:16,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:16,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:17,042 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:17,042 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:17,063 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:17,066 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:17,066 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:26,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:26,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:26,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:27,084 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:27,084 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:27,101 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:27,105 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:27,105 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:36,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:36,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:36,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:37,115 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:37,115 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:37,131 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:37,134 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:37,134 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:46,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:46,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:46,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:47,145 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:47,145 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:47,161 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:47,164 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:47,164 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:56,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:56,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:11:56,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:11:57,174 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:11:57,176 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:11:57,197 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:11:57,200 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:11:57,200 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:06,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:06,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:06,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:07,211 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:07,213 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:07,233 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:07,237 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:07,239 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:16,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:16,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:16,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:17,250 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:17,250 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:17,269 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:17,270 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:17,270 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:26,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:26,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:26,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:27,283 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:27,283 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:27,302 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:27,305 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:27,305 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:36,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:36,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:36,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:37,318 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:37,318 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:37,335 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:37,338 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:37,338 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:46,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:46,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:46,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:47,351 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:47,351 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:47,391 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:47,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:47,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:56,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:56,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:12:56,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:12:57,404 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.04 % 2021-12-16 02:12:57,404 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 3.04 2021-12-16 02:12:57,423 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:12:57,426 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:12:57,426 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:06,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.19 % 2021-12-16 02:13:06,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:06,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:07,437 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:07,437 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:07,451 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:07,453 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:07,453 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:16,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:16,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:17,467 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:17,467 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:17,486 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:17,486 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:17,487 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:26,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:26,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:26,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:27,499 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:27,499 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:27,517 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:27,520 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:27,520 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:36,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:36,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:36,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:37,535 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:37,536 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:37,555 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:37,558 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:37,558 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:46,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:46,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:46,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:47,582 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:47,582 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:47,601 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:47,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:47,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:56,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:56,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:13:56,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:13:57,615 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:13:57,615 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:13:57,632 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:13:57,635 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:13:57,635 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:06,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:06,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:06,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:07,647 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:07,647 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:07,665 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:07,667 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:07,667 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:16,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:16,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:16,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:17,681 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:17,681 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:17,698 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:17,701 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:17,701 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:26,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:26,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:26,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:27,713 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:27,713 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:27,730 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:27,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:27,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:36,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:36,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:36,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:37,746 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:37,746 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:37,769 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:37,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:37,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:46,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:46,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:46,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:47,785 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:47,786 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:47,806 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:47,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:47,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:56,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:56,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:14:56,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:14:57,822 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:14:57,822 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:14:57,838 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:14:57,841 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:14:57,841 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:06,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:06,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:06,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:07,851 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:07,851 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:07,868 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:07,872 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:07,872 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:16,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:16,530 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:16,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:17,884 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:17,884 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:17,901 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:17,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:17,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:26,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:26,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:26,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:27,914 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:27,915 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:27,933 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:27,936 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:27,937 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:36,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:36,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:36,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:37,947 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:37,947 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:37,966 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:37,969 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:37,969 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:46,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:46,612 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:46,612 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:47,980 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:47,982 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:48,002 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:48,003 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:48,003 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:56,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:56,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:15:56,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:15:58,019 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:15:58,019 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:15:58,039 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:15:58,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:15:58,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:06,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:16:06,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:06,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:08,056 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.56 % 2021-12-16 02:16:08,056 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 4.56 2021-12-16 02:16:08,073 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:08,075 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:08,075 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:16,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.71 % 2021-12-16 02:16:16,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:16,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:18,088 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.71 % 2021-12-16 02:16:18,088 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 5.71 2021-12-16 02:16:18,105 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:18,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:18,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:26,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:26,693 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:26,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:28,118 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:28,118 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:16:28,138 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:28,141 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:28,141 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:36,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:36,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:36,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:38,154 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:38,154 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:16:38,171 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:38,173 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:38,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:46,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:46,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:46,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:48,187 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:48,187 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:16:48,207 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:48,210 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:48,210 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:56,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:56,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:16:56,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:16:58,221 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:16:58,221 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:16:58,240 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:16:58,243 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:16:58,243 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:06,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:06,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:06,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:08,256 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:08,256 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:08,279 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:08,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:08,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:16,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:16,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:16,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:18,293 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:18,293 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:18,312 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:18,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:18,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:26,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:26,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:26,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:28,336 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:28,336 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:28,357 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:28,361 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:28,361 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:36,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:36,862 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:36,862 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:38,374 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:38,374 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:38,400 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:38,403 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:38,403 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:46,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:46,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:46,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:48,418 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:48,418 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:48,444 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:48,446 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:48,447 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:56,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:56,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:17:56,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:17:58,457 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:17:58,459 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:17:58,483 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:17:58,484 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:17:58,484 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:06,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:06,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:06,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:08,496 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:08,496 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:08,516 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:08,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:08,517 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:16,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:16,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:16,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:18,528 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:18,528 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:18,546 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:18,549 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:18,551 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:26,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:26,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:26,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:28,562 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:28,562 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:28,581 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:28,584 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:28,584 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:36,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:36,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:36,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:38,595 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:38,595 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:38,614 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:38,617 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:38,617 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:46,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:47,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:47,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:48,627 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:48,627 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:48,644 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:48,645 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:48,647 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:57,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:57,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:18:57,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:18:58,657 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:18:58,657 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:18:58,674 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:18:58,676 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:18:58,677 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:07,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:07,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:07,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:08,688 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:08,690 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:19:08,707 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:08,710 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:08,710 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:17,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:17,061 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:17,061 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:18,723 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:18,723 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:19:18,743 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:18,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:18,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:27,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:27,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:27,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:28,759 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.09 % 2021-12-16 02:19:28,759 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 6.09 2021-12-16 02:19:28,778 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:28,779 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:28,779 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:37,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:37,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:37,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:38,796 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:38,796 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:19:38,812 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:38,815 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:38,816 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:47,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:47,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:47,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:48,826 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:48,828 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:19:48,848 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:48,849 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:48,851 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:57,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:57,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:19:57,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:19:58,862 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:19:58,862 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:19:58,881 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:19:58,884 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:19:58,885 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:07,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:07,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:07,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:08,895 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:08,895 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:08,914 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:08,917 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:08,917 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:17,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:17,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:17,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:18,927 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:18,927 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:18,944 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:18,947 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:18,947 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:27,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:27,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:27,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:28,959 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:28,959 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:28,976 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:28,979 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:28,980 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:37,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:37,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:37,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:38,990 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:38,990 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:39,006 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:39,009 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:39,009 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:47,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:47,244 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:47,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:49,022 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:49,022 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:49,042 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:49,045 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:49,046 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:57,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:57,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:20:57,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:20:59,056 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:20:59,056 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:20:59,082 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:20:59,084 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:20:59,085 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:07,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:07,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:07,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:09,095 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:09,095 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:09,115 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:09,118 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:09,118 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:17,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:17,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:17,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:19,130 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:19,130 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:19,148 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:19,150 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:19,150 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:27,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:27,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:27,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:29,161 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:29,161 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:29,180 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:29,181 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:29,181 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:37,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:37,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:37,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:39,204 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:39,204 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:39,223 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:39,226 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:39,226 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:47,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:47,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:47,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:49,236 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:49,236 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:49,259 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:49,262 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:49,262 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:57,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:57,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:21:57,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:21:59,289 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:21:59,289 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:21:59,305 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:21:59,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:21:59,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:07,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:07,400 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:07,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:09,319 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:09,319 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:22:09,338 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:09,339 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:09,339 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:17,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:17,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:17,423 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:19,351 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:19,351 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:22:19,371 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:19,372 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:19,374 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:27,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:27,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:27,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:29,384 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:29,384 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:22:29,404 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:29,405 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:29,405 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:37,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:37,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:37,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:39,417 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 02:22:39,417 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 7.61 2021-12-16 02:22:39,436 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:39,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:39,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:47,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 02:22:47,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:47,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:49,450 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 02:22:49,450 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 02:22:49,473 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:49,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:49,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:57,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 02:22:57,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:22:57,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:22:59,489 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 02:22:59,490 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 02:22:59,506 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:22:59,509 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:22:59,509 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:07,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:07,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:07,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:09,523 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:09,525 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:09,543 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:09,549 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:09,549 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:17,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:17,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:17,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:19,559 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:19,559 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:19,576 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:19,578 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:19,578 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:27,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:27,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:27,592 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:29,589 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:29,589 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:29,605 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:29,608 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:29,608 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:37,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:37,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:37,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:39,618 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:39,618 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:39,637 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:39,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:39,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:47,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:47,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:47,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:49,651 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:49,651 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:49,674 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:49,678 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:49,678 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:57,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:57,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:23:57,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:23:59,688 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:23:59,690 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:23:59,706 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:23:59,709 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:23:59,709 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:07,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:07,683 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:07,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:09,720 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:09,721 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:09,743 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:09,744 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:09,744 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:17,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:17,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:17,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:19,755 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:19,756 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:19,773 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:19,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:19,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:27,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:27,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:27,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:29,786 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:29,786 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:29,803 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:29,805 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:29,805 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:37,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:37,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:37,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:39,818 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:39,819 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:39,849 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:39,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:39,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:47,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:47,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:47,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:49,865 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:49,865 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:49,882 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:49,884 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:49,885 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:57,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:57,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:24:57,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:24:59,897 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:24:59,897 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:24:59,917 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:24:59,920 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:24:59,920 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:07,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:07,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:07,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:09,931 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:09,933 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:25:09,950 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:25:09,954 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:25:09,954 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:17,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:17,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:17,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:19,967 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:19,967 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:25:19,987 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:25:19,990 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:25:19,990 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:27,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:27,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:27,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:30,002 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:30,003 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:25:30,025 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:25:30,029 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:25:30,029 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:37,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:37,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:37,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:40,040 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:40,040 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:25:40,061 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:25:40,065 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:25:40,065 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:47,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:47,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:47,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:50,075 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:50,075 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:25:50,092 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:25:50,094 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:25:50,094 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:25:57,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:25:57,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:25:57,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:00,105 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 02:26:00,105 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 02:26:00,121 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:00,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:00,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:07,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.27 % 2021-12-16 02:26:07,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:07,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:10,134 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.27 % 2021-12-16 02:26:10,134 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.27 2021-12-16 02:26:10,157 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:10,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:10,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:17,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.27 % 2021-12-16 02:26:17,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:17,938 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:20,170 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.27 % 2021-12-16 02:26:20,170 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.27 2021-12-16 02:26:20,188 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:20,193 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:20,193 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:27,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:27,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:27,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:30,203 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:30,203 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:26:30,223 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:30,224 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:30,224 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:37,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:37,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:37,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:40,236 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:40,236 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:26:40,253 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:40,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:40,255 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:47,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:47,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:47,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:50,265 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:50,265 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:26:50,282 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:26:50,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:26:50,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:26:58,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:26:58,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:26:58,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:00,295 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:00,295 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:00,309 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:00,312 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:00,312 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:08,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:08,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:08,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:10,325 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:10,325 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:10,348 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:10,352 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:10,352 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:18,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:18,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:18,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:20,364 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:20,364 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:20,381 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:20,384 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:20,384 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:28,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:28,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:28,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:30,398 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:30,398 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:30,417 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:30,420 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:30,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:38,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:38,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:38,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:40,433 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:40,433 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:40,460 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:40,464 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:40,464 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:48,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:48,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:48,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:50,476 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:50,476 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:27:50,493 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:27:50,496 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:27:50,496 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:27:58,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:27:58,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:27:58,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:00,509 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:00,509 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:00,528 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:00,530 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:00,530 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:08,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:08,163 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:08,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:10,542 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:10,542 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:10,562 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:10,565 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:10,565 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:18,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:18,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:18,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:20,578 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:20,578 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:20,598 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:20,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:20,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:28,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:28,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:28,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:30,612 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:30,612 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:30,630 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:30,631 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:30,631 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:38,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:38,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:38,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:40,641 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:40,642 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:40,657 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:40,660 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:40,660 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:48,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:48,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:48,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:50,670 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:50,671 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:28:50,687 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:28:50,690 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:28:50,690 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:28:58,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:28:58,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:28:58,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:00,706 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:29:00,706 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:29:00,723 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:00,724 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:00,724 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:08,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:29:08,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:08,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:10,734 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:29:10,736 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:29:10,752 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:10,753 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:10,753 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:18,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:29:18,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:18,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:20,766 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.65 % 2021-12-16 02:29:20,766 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 10.65 2021-12-16 02:29:20,783 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:20,786 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:20,786 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:28,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.79 % 2021-12-16 02:29:28,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:28,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:30,799 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.79 % 2021-12-16 02:29:30,799 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 11.79 2021-12-16 02:29:30,821 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:30,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:30,822 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:38,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:29:38,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:38,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:40,834 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:29:40,834 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:29:40,851 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:40,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:40,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:48,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:29:48,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:48,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:50,864 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:29:50,865 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:29:50,884 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:29:50,887 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:29:50,887 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:29:58,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:29:58,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:29:58,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:00,897 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:00,898 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:00,917 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:00,918 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:00,918 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:08,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:08,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:08,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:10,931 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:10,931 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:10,950 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:10,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:10,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:18,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:18,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:18,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:20,963 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:20,964 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:20,987 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:20,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:20,990 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:28,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:28,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:28,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:31,000 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:31,000 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:31,023 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:31,025 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:31,025 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:38,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:38,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:38,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:41,036 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:41,036 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:41,055 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:41,056 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:41,058 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:48,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:48,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:48,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:51,066 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:51,068 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:30:51,084 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:30:51,086 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:30:51,086 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:30:58,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:30:58,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:30:58,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:01,098 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:01,098 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:01,115 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:01,118 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:01,118 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:08,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:08,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:08,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:11,130 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:11,130 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:11,148 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:11,151 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:11,153 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:18,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:18,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:18,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:21,163 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:21,164 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:21,183 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:21,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:21,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:28,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:28,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:28,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:31,197 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:31,197 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:31,219 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:31,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:31,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:38,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:38,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:38,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:41,233 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:41,233 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:41,252 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:41,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:41,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:48,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:48,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:48,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:51,263 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:51,263 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:31:51,279 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:31:51,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:31:51,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:31:58,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:31:58,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:31:58,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:01,293 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:01,293 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:32:01,312 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:01,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:01,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:08,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:08,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:08,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:11,325 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:11,325 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:32:11,345 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:11,348 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:11,348 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:18,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:18,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:18,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:21,361 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:21,361 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:32:21,378 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:21,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:21,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:28,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:28,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:28,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:31,394 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.17 % 2021-12-16 02:32:31,394 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.17 2021-12-16 02:32:31,411 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:31,413 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:31,414 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:38,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.55 % 2021-12-16 02:32:38,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:38,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:41,424 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.93 % 2021-12-16 02:32:41,426 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 12.93 2021-12-16 02:32:41,441 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:41,444 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:41,444 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:48,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.31 % 2021-12-16 02:32:48,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:48,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:51,457 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.31 % 2021-12-16 02:32:51,457 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.31 2021-12-16 02:32:51,477 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:32:51,482 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:32:51,482 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:32:58,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.31 % 2021-12-16 02:32:58,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:32:58,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:01,496 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:01,497 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:01,515 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:01,517 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:01,517 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:08,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:08,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:08,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:11,529 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:11,529 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:11,553 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:11,555 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:11,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:18,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:18,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:18,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:21,585 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:21,585 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:21,605 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:21,608 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:21,608 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:28,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:28,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:28,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:31,621 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:31,621 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:31,641 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:31,642 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:31,642 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:38,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:38,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:38,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:41,657 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:41,657 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:41,680 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:41,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:41,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:48,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:48,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:48,857 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:51,711 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:51,713 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:33:51,730 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:33:51,732 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:33:51,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:33:58,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:33:58,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:33:58,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:01,743 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:01,744 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:01,767 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:01,770 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:01,770 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:08,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:08,895 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:08,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:11,782 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:11,782 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:11,799 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:11,802 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:11,802 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:18,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:18,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:18,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:21,816 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:21,816 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:21,842 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:21,846 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:21,846 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:28,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:28,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:28,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:31,857 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:31,858 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:31,874 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:31,875 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:31,875 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:38,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:38,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:38,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:41,887 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:41,887 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:41,905 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:41,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:41,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:48,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:48,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:48,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:51,921 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:51,923 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:34:51,941 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:34:51,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:34:51,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:34:58,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:34:58,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:34:58,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:01,956 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:01,956 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:01,971 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:01,973 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:01,973 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:08,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:09,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:09,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:11,986 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:11,986 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:12,002 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:12,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:12,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:19,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:19,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:19,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:22,016 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:22,016 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:22,033 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:22,035 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:22,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:29,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:29,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:29,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:32,045 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:32,046 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:32,065 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:32,068 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:32,068 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:39,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:39,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:39,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:42,078 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:42,078 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:42,095 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:42,099 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:42,099 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:49,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:49,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:49,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:52,111 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.69 % 2021-12-16 02:35:52,111 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 13.69 2021-12-16 02:35:52,127 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:35:52,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:35:52,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:35:59,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.45 % 2021-12-16 02:35:59,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:35:59,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:02,138 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.45 % 2021-12-16 02:36:02,140 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 14.45 2021-12-16 02:36:02,157 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:02,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:02,160 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:09,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 02:36:09,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:09,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:12,168 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 02:36:12,168 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 14.83 2021-12-16 02:36:12,186 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:12,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:12,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:19,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:19,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:19,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:22,197 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:22,197 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:36:22,213 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:22,216 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:22,216 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:29,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:29,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:29,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:32,229 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:32,229 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:36:32,249 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:32,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:32,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:39,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:39,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:39,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:42,263 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:42,263 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:36:42,280 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:42,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:42,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:49,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:49,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:49,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:52,293 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:52,293 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:36:52,311 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:36:52,312 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:36:52,312 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:36:59,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:36:59,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:36:59,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:02,323 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:02,323 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:02,341 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:02,344 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:02,344 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:09,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:09,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:09,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:12,355 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:12,355 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:12,371 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:12,374 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:12,375 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:19,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:19,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:19,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:22,385 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:22,385 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:22,403 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:22,405 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:22,405 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:29,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:29,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:29,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:32,415 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:32,415 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:32,431 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:32,434 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:32,434 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:39,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:39,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:39,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:42,444 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:42,444 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:42,463 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:42,464 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:42,464 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:49,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:49,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:49,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:52,480 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:52,480 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:37:52,496 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:37:52,500 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:37:52,500 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:37:59,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:37:59,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:37:59,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:02,510 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:02,512 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:02,528 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:02,529 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:02,529 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:09,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:09,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:09,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:12,542 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:12,542 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:12,565 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:12,568 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:12,569 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:19,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:19,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:19,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:22,582 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:22,582 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:22,601 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:22,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:22,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:29,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:29,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:29,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:32,614 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:32,615 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:32,637 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:32,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:32,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:39,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:39,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:39,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:42,678 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:42,680 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:42,698 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:42,701 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:42,701 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:49,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:49,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:49,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:52,714 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:52,714 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:38:52,732 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:38:52,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:38:52,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:38:59,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:38:59,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:38:59,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:02,747 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:39:02,747 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.21 2021-12-16 02:39:02,773 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:02,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:02,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:09,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.21 % 2021-12-16 02:39:09,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:09,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:12,786 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.59 % 2021-12-16 02:39:12,788 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 15.59 2021-12-16 02:39:12,803 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:12,805 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:12,805 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:19,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 02:39:19,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:19,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:22,816 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 02:39:22,818 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.35 2021-12-16 02:39:22,834 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:22,836 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:22,836 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:29,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 02:39:29,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:29,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:32,848 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.35 % 2021-12-16 02:39:32,848 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.35 2021-12-16 02:39:32,888 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:32,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:32,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:39,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:39:39,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:39,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:42,904 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:39:42,904 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:39:42,947 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:42,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:42,957 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:49,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:39:49,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:49,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:52,969 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:39:52,970 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:39:52,984 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:39:52,986 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:39:52,986 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:39:59,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:39:59,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:39:59,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:03,002 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:03,003 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:03,023 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:03,026 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:03,028 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:09,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:09,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:09,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:13,038 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:13,038 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:13,055 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:13,056 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:13,056 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:19,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:19,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:19,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:23,066 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:23,068 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:23,085 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:23,086 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:23,086 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:29,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:29,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:29,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:33,098 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:33,098 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:33,118 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:33,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:33,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:39,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:39,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:39,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:43,131 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:43,131 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:43,151 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:43,154 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:43,154 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:49,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:49,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:49,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:53,164 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:53,164 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:40:53,180 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:40:53,184 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:40:53,184 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:40:59,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:40:59,668 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:40:59,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:03,203 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:03,203 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:03,226 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:03,247 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:03,247 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:09,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:09,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:09,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:13,257 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:13,257 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:13,275 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:13,278 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:13,278 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:19,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:19,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:19,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:23,289 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:23,289 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:23,309 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:23,311 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:23,311 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:29,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:29,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:29,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:33,321 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:33,321 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:33,338 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:33,339 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:33,339 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:39,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:39,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:39,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:43,351 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:43,351 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:43,374 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:43,375 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:43,375 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:49,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:49,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:49,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:53,388 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:53,388 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:41:53,407 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:41:53,410 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:41:53,410 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:41:59,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:41:59,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:41:59,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:03,421 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:42:03,421 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:42:03,438 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:03,440 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:03,440 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:09,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:42:09,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:09,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:13,450 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:42:13,451 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 16.74 2021-12-16 02:42:13,473 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:13,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:13,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:19,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.74 % 2021-12-16 02:42:19,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:19,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:23,487 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-16 02:42:23,489 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 17.11 2021-12-16 02:42:23,506 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:23,509 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:23,509 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:29,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-16 02:42:29,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:29,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:33,520 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-16 02:42:33,520 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 17.11 2021-12-16 02:42:33,539 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:33,540 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:33,540 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:39,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.50 % 2021-12-16 02:42:39,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:39,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:43,552 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.87 % 2021-12-16 02:42:43,552 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 17.87 2021-12-16 02:42:43,571 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:43,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:43,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:49,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.87 % 2021-12-16 02:42:49,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:49,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:53,584 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.87 % 2021-12-16 02:42:53,584 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 17.87 2021-12-16 02:42:53,599 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:42:53,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:42:53,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:42:59,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.87 % 2021-12-16 02:42:59,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:42:59,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:03,612 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.87 % 2021-12-16 02:43:03,614 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 17.87 2021-12-16 02:43:03,631 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:03,632 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:03,634 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:09,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:09,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:43:09,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:13,644 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:13,644 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:43:13,661 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:13,665 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:13,665 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:19,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:19,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:43:19,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:23,678 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:23,678 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:43:23,697 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:23,700 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:23,700 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:29,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:29,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:43:29,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:33,710 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:33,710 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:43:33,727 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:33,730 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:33,730 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:39,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:39,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:43:39,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:43,740 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:43,740 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:43:43,756 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:43,759 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:43,759 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:50,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:50,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:43:50,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:43:53,769 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:43:53,769 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:43:53,793 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:43:53,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:43:53,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:00,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:00,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:00,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:03,805 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:03,805 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:03,822 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:03,825 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:03,825 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:10,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:10,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:10,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:13,836 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:13,836 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:13,854 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:13,855 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:13,855 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:20,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:20,079 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:20,079 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:23,868 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:23,868 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:23,901 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:23,905 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:23,905 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:30,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:30,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:30,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:33,915 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:33,917 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:33,934 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:33,937 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:33,937 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:40,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:40,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:40,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:43,948 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:43,948 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:43,973 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:43,976 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:43,976 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:50,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:50,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:44:50,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:44:53,989 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:44:53,989 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:44:54,007 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:44:54,010 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:44:54,010 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:00,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:00,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:00,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:04,022 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:04,022 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:45:04,039 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:04,040 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:04,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:10,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:10,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:10,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:14,065 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:14,065 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:45:14,088 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:14,092 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:14,092 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:20,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:20,194 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:20,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:24,102 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:24,102 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.26 2021-12-16 02:45:24,119 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:24,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:24,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:30,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.26 % 2021-12-16 02:45:30,216 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:30,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:34,134 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.64 % 2021-12-16 02:45:34,134 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.64 2021-12-16 02:45:34,157 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:34,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:34,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:40,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.64 % 2021-12-16 02:45:40,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:40,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:44,168 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.64 % 2021-12-16 02:45:44,168 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 18.64 2021-12-16 02:45:44,187 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:44,203 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:44,203 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:50,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.02 % 2021-12-16 02:45:50,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:45:50,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:45:54,213 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-16 02:45:54,213 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.4 2021-12-16 02:45:54,229 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:45:54,230 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:45:54,230 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:00,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-16 02:46:00,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:00,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:04,242 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-16 02:46:04,242 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.4 2021-12-16 02:46:04,259 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:04,260 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:04,262 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:10,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-16 02:46:10,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:10,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:14,276 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.40 % 2021-12-16 02:46:14,276 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.4 2021-12-16 02:46:14,296 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:14,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:14,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:20,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:20,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:20,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:24,311 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:24,312 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:46:24,335 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:24,336 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:24,336 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:30,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:30,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:30,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:34,351 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:34,351 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:46:34,372 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:34,377 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:34,377 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:40,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:40,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:40,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:44,388 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:44,388 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:46:44,407 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:44,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:44,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:50,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:50,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:46:50,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:46:54,423 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:46:54,423 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:46:54,438 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:46:54,440 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:46:54,441 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:00,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:00,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:00,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:04,453 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:04,453 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:04,473 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:04,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:04,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:10,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:10,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:10,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:14,487 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:14,487 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:14,509 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:14,510 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:14,510 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:20,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:20,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:20,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:24,523 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:24,523 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:24,543 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:24,546 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:24,546 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:30,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:30,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:30,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:34,555 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:34,555 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:34,572 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:34,573 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:34,573 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:40,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:40,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:40,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:44,584 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:44,585 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:44,601 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:44,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:44,602 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:50,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:50,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:47:50,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:47:54,615 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:47:54,615 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:47:54,637 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:47:54,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:47:54,641 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:00,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:00,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:00,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:04,651 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:04,651 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:48:04,667 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:04,670 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:04,670 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:10,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:10,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:10,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:14,683 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:14,683 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:48:14,707 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:14,710 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:14,710 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:20,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:20,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:20,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:24,723 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:24,723 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:48:24,740 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:24,743 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:24,743 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:30,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:30,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:30,595 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:34,755 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:34,755 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:48:34,775 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:34,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:34,778 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:40,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:40,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:40,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:44,789 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.77 % 2021-12-16 02:48:44,789 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 19.77 2021-12-16 02:48:44,806 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:44,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:44,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:50,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.16 % 2021-12-16 02:48:50,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:48:50,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:48:54,819 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.16 % 2021-12-16 02:48:54,819 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 20.16 2021-12-16 02:48:54,836 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:48:54,839 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:48:54,839 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:00,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.16 % 2021-12-16 02:49:00,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:00,645 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:04,851 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.92 % 2021-12-16 02:49:04,852 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 20.92 2021-12-16 02:49:04,869 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:04,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:04,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:10,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.92 % 2021-12-16 02:49:10,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:10,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:14,881 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.92 % 2021-12-16 02:49:14,881 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 20.92 2021-12-16 02:49:14,901 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:14,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:14,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:20,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.92 % 2021-12-16 02:49:20,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:20,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:24,915 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.92 % 2021-12-16 02:49:24,915 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 20.92 2021-12-16 02:49:24,936 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:24,938 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:24,940 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:30,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:30,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:30,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:34,950 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:34,950 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:49:34,971 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:34,973 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:34,973 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:40,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:40,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:40,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:44,983 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:44,983 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:49:45,000 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:45,003 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:45,003 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:50,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:50,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:49:50,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:49:55,013 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:49:55,015 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:49:55,035 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:49:55,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:49:55,036 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:00,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:00,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:00,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:05,049 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:05,049 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:05,068 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:05,071 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:05,072 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:10,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:10,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:10,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:15,082 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:15,082 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:15,098 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:15,101 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:15,101 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:20,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:20,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:20,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:25,111 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:25,111 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:25,128 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:25,130 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:25,130 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:30,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:30,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:30,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:35,144 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:35,144 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:35,161 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:35,164 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:35,164 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:40,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:40,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:40,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:45,176 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:45,176 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:45,203 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:45,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:45,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:50,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:50,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:50:50,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:50:55,217 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:50:55,217 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:50:55,236 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:50:55,240 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:50:55,240 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:00,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:00,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:00,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:05,250 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:05,250 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:51:05,265 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:05,266 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:05,266 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:10,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:10,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:10,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:15,276 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:15,278 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:51:15,295 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:15,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:15,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:20,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:20,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:20,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:25,312 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:25,312 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:51:25,332 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:25,336 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:25,336 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:30,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:30,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:30,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:35,346 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:35,348 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:51:35,364 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:35,365 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:35,365 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:40,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:40,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:40,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:45,377 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.30 % 2021-12-16 02:51:45,377 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.3 2021-12-16 02:51:45,392 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:45,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:45,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:50,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.68 % 2021-12-16 02:51:51,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:51:51,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:51:55,404 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.68 % 2021-12-16 02:51:55,404 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.68 2021-12-16 02:51:55,420 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:51:55,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:51:55,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:01,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.68 % 2021-12-16 02:52:01,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:01,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:05,433 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.68 % 2021-12-16 02:52:05,433 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 21.68 2021-12-16 02:52:05,450 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:05,453 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:05,453 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:11,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.06 % 2021-12-16 02:52:11,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:11,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:15,463 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:15,463 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.44 2021-12-16 02:52:15,483 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:15,486 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:15,486 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:21,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:21,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:21,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:25,496 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:25,496 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.44 2021-12-16 02:52:25,513 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:25,515 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:25,515 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:31,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:31,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:31,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:35,526 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:35,526 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.44 2021-12-16 02:52:35,546 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:35,549 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:35,549 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:41,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 02:52:41,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:41,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:45,559 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:52:45,559 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:52:45,579 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:45,582 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:45,582 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:51,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:52:51,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:52:51,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:52:55,594 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:52:55,594 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:52:55,621 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:52:55,642 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:52:55,644 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:01,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:01,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:01,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:05,657 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:05,657 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:05,678 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:05,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:05,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:11,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:11,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:11,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:15,693 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:15,694 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:15,711 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:15,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:15,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:21,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:21,158 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:21,158 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:25,727 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:25,729 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:25,753 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:25,756 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:25,756 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:31,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:31,178 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:31,178 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:35,767 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:35,772 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:35,793 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:35,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:35,798 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:41,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:41,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:41,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:45,809 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:45,809 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:45,835 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:45,836 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:45,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:51,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:51,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:53:51,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:53:55,849 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:53:55,849 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:53:55,868 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:53:55,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:53:55,871 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:01,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:01,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:01,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:05,881 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:05,881 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:05,898 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:05,900 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:05,900 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:11,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:11,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:11,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:15,911 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:15,913 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:15,931 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:15,934 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:15,934 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:21,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:21,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:21,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:25,947 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:25,947 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:25,967 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:25,970 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:25,970 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:31,288 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:31,301 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:31,301 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:35,982 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:35,982 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:35,997 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:36,000 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:36,000 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:41,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:41,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:41,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:46,010 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:46,010 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:46,029 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:46,032 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:46,032 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:51,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:51,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:54:51,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:54:56,042 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.82 % 2021-12-16 02:54:56,042 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 22.82 2021-12-16 02:54:56,059 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:54:56,061 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:54:56,061 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:01,344 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.20 % 2021-12-16 02:55:01,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:01,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:06,072 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.20 % 2021-12-16 02:55:06,072 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 23.2 2021-12-16 02:55:06,092 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:06,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:06,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:11,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.20 % 2021-12-16 02:55:11,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:11,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:16,105 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.58 % 2021-12-16 02:55:16,105 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 23.58 2021-12-16 02:55:16,121 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:16,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:16,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:21,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.96 % 2021-12-16 02:55:21,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:21,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:26,134 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:26,134 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:55:26,151 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:26,153 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:26,153 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:31,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:31,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:31,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:36,164 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:36,165 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:55:36,184 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:36,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:36,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:41,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:41,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:41,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:46,198 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:46,200 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:55:46,217 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:46,220 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:46,220 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:51,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:51,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:55:51,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:55:56,232 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:55:56,232 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:55:56,253 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:55:56,256 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:55:56,256 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:01,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:01,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:01,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:06,273 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:06,273 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:06,292 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:06,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:06,298 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:11,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:11,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:11,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:16,309 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:16,309 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:16,328 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:16,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:16,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:21,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:21,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:21,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:26,339 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:26,341 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:26,358 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:26,361 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:26,361 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:31,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:31,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:31,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:36,371 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:36,371 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:36,388 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:36,390 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:36,390 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:41,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:41,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:41,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:46,401 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:46,403 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:46,426 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:46,428 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:46,428 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:51,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:51,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:56:51,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:56:56,457 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:56:56,457 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:56:56,483 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:56:56,487 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:56:56,487 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:01,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:01,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:01,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:06,497 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:06,497 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:06,513 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:06,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:06,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:11,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:11,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:11,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:16,529 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:16,529 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:16,563 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:16,568 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:16,568 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:21,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:21,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:21,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:26,578 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:26,578 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:26,595 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:26,596 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:26,596 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:31,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:31,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:31,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:36,608 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:36,608 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:36,625 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:36,627 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:36,627 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:41,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:41,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:41,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:46,637 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:46,637 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:46,654 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:46,657 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:46,657 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:51,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:51,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:57:51,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:57:56,667 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:57:56,667 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:57:56,684 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:57:56,687 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:57:56,687 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:01,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:01,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:01,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:06,698 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:06,698 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:58:06,717 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:06,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:06,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:11,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:11,696 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:11,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:16,730 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:16,730 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:58:16,746 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:16,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:16,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:21,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:21,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:21,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:26,763 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:26,763 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:58:26,785 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:26,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:26,789 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:31,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:31,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:31,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:36,799 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.34 % 2021-12-16 02:58:36,799 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 24.34 2021-12-16 02:58:36,818 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:36,821 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:36,821 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:41,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.10 % 2021-12-16 02:58:41,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:41,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:46,832 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:58:46,832 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:58:46,851 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:46,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:46,854 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:51,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:58:51,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:58:51,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:58:56,865 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:58:56,865 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:58:56,887 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:58:56,888 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:58:56,888 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:01,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:01,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:01,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:06,900 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:06,901 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:06,938 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:06,941 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:06,941 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:11,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:11,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:11,831 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:16,953 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:16,953 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:16,973 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:16,976 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:16,976 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:21,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:21,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:21,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:26,987 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:26,987 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:27,007 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:27,010 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:27,010 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:31,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:31,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:31,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:37,022 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:37,023 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:37,039 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:37,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:37,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:41,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:41,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:41,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:47,052 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:47,053 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:47,073 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:47,075 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:47,075 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:51,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:51,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 02:59:51,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 02:59:57,088 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 02:59:57,088 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 02:59:57,105 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 02:59:57,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 02:59:57,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:01,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:01,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:01,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:07,121 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:07,121 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:07,144 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:07,145 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:07,147 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:11,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:11,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:11,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:17,157 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:17,157 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:17,174 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:17,178 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:17,178 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:21,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:22,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:22,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:27,188 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:27,188 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:27,204 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:27,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:27,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:32,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:32,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:32,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:37,217 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:37,217 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:37,240 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:37,242 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:37,242 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:42,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:42,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:42,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:47,259 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:47,259 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:47,278 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:47,280 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:47,280 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:52,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:52,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:00:52,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:00:57,290 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:00:57,292 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:00:57,308 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:00:57,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:00:57,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:02,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:02,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:02,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:07,321 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:07,321 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:01:07,345 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:07,346 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:07,346 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:12,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:12,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:12,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:17,358 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:17,358 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:01:17,375 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:17,378 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:17,378 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:22,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:22,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:22,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:27,388 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:27,388 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:01:27,405 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:27,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:27,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:32,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:32,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:32,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:37,421 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:37,421 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:01:37,440 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:37,441 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:37,443 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:42,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:42,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:42,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:47,454 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.86 % 2021-12-16 03:01:47,454 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 25.86 2021-12-16 03:01:47,474 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:47,479 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:47,479 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:52,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.62 % 2021-12-16 03:01:52,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:01:52,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:01:57,487 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.99 % 2021-12-16 03:01:57,489 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 26.99 2021-12-16 03:01:57,503 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:01:57,506 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:01:57,506 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:02,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:02,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:02,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:07,516 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:07,516 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:07,536 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:07,538 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:07,538 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:12,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:12,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:12,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:17,549 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:17,549 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:17,565 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:17,568 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:17,568 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:22,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:22,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:22,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:27,581 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:27,582 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:27,601 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:27,605 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:27,605 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:32,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:32,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:32,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:37,615 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:37,617 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:37,638 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:37,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:37,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:42,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:42,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:42,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:47,657 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:47,657 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:47,677 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:47,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:47,683 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:52,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:52,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:02:52,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:02:57,691 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:02:57,693 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:02:57,710 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:02:57,711 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:02:57,711 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:02,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:02,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:02,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:07,723 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:07,723 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:07,744 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:07,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:07,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:12,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:12,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:12,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:17,759 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:17,759 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:17,779 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:17,782 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:17,782 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:22,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:22,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:22,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:27,793 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:27,793 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:27,811 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:27,812 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:27,812 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:32,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:32,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:32,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:37,842 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:37,842 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:37,861 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:37,862 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:37,862 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:42,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:42,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:42,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:47,875 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:47,875 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:47,891 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:47,892 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:47,892 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:52,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:52,420 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:03:52,420 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:03:57,944 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:03:57,944 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:03:57,963 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:03:57,966 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:03:57,966 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:02,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:02,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:02,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:07,980 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:07,980 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:08,003 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:08,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:08,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:12,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:12,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:12,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:18,017 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:18,019 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:18,036 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:18,038 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:18,038 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:22,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:22,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:22,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:28,058 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:28,058 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:28,098 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:28,102 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:28,102 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:32,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:32,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:32,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:38,114 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:38,114 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:38,134 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:38,135 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:38,135 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:42,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:42,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:42,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:48,147 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:48,148 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:48,171 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:48,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:48,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:52,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:52,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:04:52,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:04:58,184 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.38 % 2021-12-16 03:04:58,184 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 27.38 2021-12-16 03:04:58,203 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:04:58,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:04:58,206 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:02,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.14 % 2021-12-16 03:05:02,568 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:02,568 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:08,217 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:08,217 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:08,233 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:08,236 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:08,236 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:12,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:12,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:12,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:18,246 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:18,247 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:18,272 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:18,276 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:18,276 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:22,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:22,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:22,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:28,289 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:28,289 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:28,306 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:28,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:28,309 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:32,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:32,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:32,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:38,322 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:38,322 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:38,341 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:38,344 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:38,344 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:42,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:42,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:42,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:48,357 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:48,358 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:48,381 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:48,385 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:48,385 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:52,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:52,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:05:52,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:05:58,397 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:05:58,397 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:05:58,415 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:05:58,417 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:05:58,417 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:02,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:02,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:02,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:08,430 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:08,430 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:08,457 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:08,460 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:08,460 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:12,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:12,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:12,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:18,473 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:18,473 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:18,490 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:18,493 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:18,493 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:22,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:22,706 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:22,706 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:28,503 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:28,503 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:28,517 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:28,519 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:28,519 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:32,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:32,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:32,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:38,532 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:38,532 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:38,551 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:38,553 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:38,555 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:42,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:42,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:42,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:48,565 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:48,565 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:48,582 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:48,585 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:48,585 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:52,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:52,770 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:06:52,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:06:58,595 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:06:58,595 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:06:58,611 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:06:58,612 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:06:58,612 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:02,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:02,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:02,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:08,624 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:08,624 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:08,641 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:08,644 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:08,644 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:12,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:12,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:12,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:18,664 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:18,664 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:18,681 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:18,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:18,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:22,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:22,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:22,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:28,693 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:28,694 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:28,710 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:28,711 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:28,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:32,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:32,846 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:32,846 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:38,724 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:38,724 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:38,743 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:38,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:38,746 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:42,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:42,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:42,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:48,759 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:48,759 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:48,780 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:48,785 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:48,785 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:52,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:52,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:07:52,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:07:58,795 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:07:58,795 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:07:58,812 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:07:58,813 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:07:58,813 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:02,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:08:02,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:02,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:08,823 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:08:08,825 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 28.9 2021-12-16 03:08:08,842 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:08,845 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:08,845 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:12,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.90 % 2021-12-16 03:08:12,927 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:12,927 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:18,855 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.66 % 2021-12-16 03:08:18,855 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 29.66 2021-12-16 03:08:18,871 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:18,872 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:18,874 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:22,938 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.04 % 2021-12-16 03:08:22,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:22,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:28,884 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:28,884 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:08:28,904 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:28,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:28,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:32,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:32,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:32,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:38,920 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:38,920 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:08:38,937 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:38,940 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:38,940 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:42,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:43,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:43,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:48,950 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:48,950 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:08:48,967 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:48,970 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:48,970 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:53,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:53,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:08:53,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:08:58,980 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:08:58,982 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:08:59,006 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:08:59,009 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:08:59,009 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:03,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:03,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:03,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:09,023 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:09,023 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:09,040 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:09,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:09,042 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:13,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:13,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:13,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:19,053 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:19,053 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:19,071 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:19,072 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:19,072 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:23,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:23,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:23,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:29,084 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:29,085 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:29,108 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:29,111 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:29,111 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:33,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:33,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:33,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:39,122 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:39,122 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:39,141 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:39,144 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:39,144 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:43,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:43,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:43,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:49,154 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:49,154 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:49,173 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:49,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:49,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:53,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:53,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:09:53,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:09:59,186 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:09:59,186 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:09:59,203 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:09:59,207 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:09:59,207 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:03,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:03,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:03,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:09,217 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:09,219 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:09,236 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:09,239 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:09,240 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:13,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:13,187 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:13,187 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:19,250 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:19,252 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:19,272 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:19,273 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:19,273 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:23,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:23,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:23,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:29,283 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:29,285 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:29,301 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:29,303 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:29,305 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:33,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:33,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:33,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:39,315 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:39,315 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:39,335 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:39,336 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:39,338 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:43,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:43,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:43,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:49,348 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:49,349 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:49,391 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:49,395 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:49,395 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:53,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:53,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:10:53,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:10:59,407 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:10:59,407 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:10:59,426 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:10:59,427 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:10:59,427 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:03,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:03,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:03,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:09,438 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:09,440 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:11:09,461 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:09,463 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:09,463 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:13,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:13,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:13,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:19,473 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:19,473 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:11:19,492 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:19,493 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:19,493 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:23,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:23,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:23,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:29,505 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:29,505 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:11:29,519 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:29,522 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:29,522 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:33,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:33,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:33,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:39,532 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.42 % 2021-12-16 03:11:39,532 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 30.42 2021-12-16 03:11:39,549 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:39,552 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:39,552 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:43,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.80 % 2021-12-16 03:11:43,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:43,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:49,562 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.18 % 2021-12-16 03:11:49,562 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.18 2021-12-16 03:11:49,579 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:49,582 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:49,582 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:53,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.56 % 2021-12-16 03:11:53,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:11:53,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:11:59,592 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:11:59,592 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:11:59,611 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:11:59,611 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:11:59,611 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:03,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:03,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:03,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:09,622 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:09,622 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:09,644 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:09,648 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:09,648 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:13,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:13,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:13,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:19,658 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:19,658 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:19,684 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:19,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:19,686 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:23,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:23,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:23,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:29,696 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:29,697 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:29,717 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:29,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:29,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:33,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:33,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:33,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:39,730 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:39,730 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:39,747 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:39,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:39,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:43,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:43,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:43,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:49,762 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:49,762 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:49,776 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:49,779 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:49,779 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:53,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:53,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:12:53,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:12:59,792 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:12:59,792 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:12:59,812 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:12:59,815 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:12:59,815 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:03,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:03,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:03,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:09,828 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:09,828 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:13:09,848 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:13:09,849 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:13:09,849 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:13,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:13,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:13,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:19,859 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:19,859 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:13:19,875 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:13:19,880 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:13:19,880 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:23,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:23,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:23,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:29,891 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:29,891 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:13:29,911 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:13:29,913 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:13:29,913 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:33,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:33,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:33,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:39,923 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:39,924 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:13:39,941 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:13:39,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:13:39,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:43,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:43,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:43,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:49,960 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:49,961 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:13:49,979 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:13:49,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:13:49,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:53,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:53,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:13:53,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:13:59,996 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:13:59,996 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:00,029 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:00,032 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:00,032 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:03,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:03,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:03,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:10,042 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:10,042 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:10,062 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:10,065 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:10,065 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:13,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:13,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:13,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:20,076 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:20,076 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:20,092 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:20,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:20,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:23,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:23,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:23,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:30,105 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:30,105 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:30,125 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:30,127 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:30,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:33,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:33,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:33,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:40,138 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:40,138 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:40,157 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:40,160 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:40,160 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:43,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:43,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:43,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:50,171 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:50,171 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 31.95 2021-12-16 03:14:50,190 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:14:50,193 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:14:50,193 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:14:53,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.95 % 2021-12-16 03:14:53,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:14:53,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:00,204 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.32 % 2021-12-16 03:15:00,204 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 32.32 2021-12-16 03:15:00,220 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:00,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:00,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:03,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.32 % 2021-12-16 03:15:03,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:03,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:10,233 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.08 % 2021-12-16 03:15:10,233 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.08 2021-12-16 03:15:10,252 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:10,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:10,253 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:13,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:13,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:13,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:20,263 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:20,265 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:15:20,286 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:20,290 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:20,290 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:23,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:23,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:23,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:30,305 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:30,305 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:15:30,323 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:30,325 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:30,325 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:33,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:33,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:33,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:40,335 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:40,336 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:15:40,357 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:40,359 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:40,359 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:43,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:43,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:43,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:50,369 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:50,371 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:15:50,388 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:15:50,391 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:15:50,391 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:15:53,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:15:53,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:15:53,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:00,401 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:00,401 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:00,423 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:00,424 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:00,424 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:03,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:03,848 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:03,848 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:10,436 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:10,437 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:10,453 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:10,457 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:10,457 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:13,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:13,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:13,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:20,467 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:20,467 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:20,486 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:20,489 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:20,489 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:23,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:23,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:23,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:30,500 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:30,500 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:30,519 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:30,522 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:30,522 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:33,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:33,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:33,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:40,532 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:40,532 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:40,551 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:40,555 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:40,555 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:43,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:43,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:43,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:50,566 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:50,566 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:16:50,582 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:16:50,584 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:16:50,584 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:16:53,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:16:53,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:16:53,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:00,595 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:00,595 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:00,615 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:00,618 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:00,618 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:03,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:03,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:03,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:10,628 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:10,630 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:10,648 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:10,653 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:10,653 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:13,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:13,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:13,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:20,663 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:20,664 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:20,680 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:20,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:20,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:24,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:24,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:24,020 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:30,693 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:30,693 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:30,710 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:30,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:30,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:34,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:34,038 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:34,038 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:40,723 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:40,724 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:40,742 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:40,743 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:40,743 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:44,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:44,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:44,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:50,763 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:50,763 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:17:50,782 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:17:50,786 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:17:50,786 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:17:54,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:17:54,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:17:54,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:00,806 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:18:00,808 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:18:00,823 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:00,826 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:00,826 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:04,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:18:04,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:04,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:10,838 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:18:10,838 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:18:10,857 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:10,859 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:10,859 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:14,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:18:14,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:14,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:20,874 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.47 % 2021-12-16 03:18:20,874 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.47 2021-12-16 03:18:20,891 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:20,894 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:20,895 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:24,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.85 % 2021-12-16 03:18:24,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:24,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:30,905 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.85 % 2021-12-16 03:18:30,907 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 33.85 2021-12-16 03:18:30,923 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:30,926 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:30,926 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:34,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.85 % 2021-12-16 03:18:34,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:34,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:40,937 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.61 % 2021-12-16 03:18:40,938 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.61 2021-12-16 03:18:40,961 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:40,963 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:40,964 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:44,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.61 % 2021-12-16 03:18:44,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:44,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:50,974 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:18:50,974 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:18:50,993 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:18:50,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:18:50,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:18:54,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:18:54,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:18:54,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:01,006 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:01,007 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:01,026 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:01,028 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:01,028 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:04,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:04,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:04,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:11,039 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:11,039 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:11,059 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:11,062 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:11,063 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:14,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:14,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:14,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:21,072 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:21,073 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:21,088 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:21,091 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:21,091 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:24,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:24,243 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:24,243 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:31,101 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:31,101 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:31,121 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:31,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:31,122 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:34,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:34,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:34,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:41,132 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:41,132 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:41,151 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:41,154 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:41,154 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:44,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:44,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:44,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:51,164 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:51,164 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:19:51,180 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:19:51,183 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:19:51,183 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:19:54,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:19:54,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:19:54,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:01,194 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:01,194 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:01,219 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:01,221 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:01,221 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:04,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:04,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:04,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:11,233 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:11,234 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:11,253 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:11,256 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:11,256 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:14,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:14,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:14,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:21,269 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:21,269 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:21,295 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:21,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:21,298 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:24,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:24,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:24,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:31,311 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:31,311 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:31,329 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:31,332 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:31,332 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:34,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:34,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:34,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:41,342 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:41,342 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:41,361 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:41,365 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:41,365 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:44,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:44,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:44,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:51,377 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:51,377 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:20:51,391 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:20:51,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:20:51,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:20:54,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:20:54,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:20:54,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:01,404 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:01,404 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:21:01,423 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:01,426 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:01,426 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:04,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:04,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:04,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:11,437 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:11,437 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:21:11,459 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:11,461 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:11,461 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:14,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:14,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:14,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:21,474 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:21,476 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:21:21,493 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:21,494 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:21,494 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:24,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:24,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:24,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:31,509 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:31,509 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 34.99 2021-12-16 03:21:31,528 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:31,532 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:31,532 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:34,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.99 % 2021-12-16 03:21:34,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:34,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:41,545 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.37 % 2021-12-16 03:21:41,545 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 35.37 2021-12-16 03:21:41,563 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:41,565 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:41,565 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:44,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.37 % 2021-12-16 03:21:44,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:44,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:51,579 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.13 % 2021-12-16 03:21:51,579 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.13 2021-12-16 03:21:51,602 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:21:51,605 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:21:51,605 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:21:54,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.13 % 2021-12-16 03:21:54,538 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:21:54,538 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:01,621 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.13 % 2021-12-16 03:22:01,621 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.13 2021-12-16 03:22:01,644 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:01,648 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:01,648 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:04,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.13 % 2021-12-16 03:22:04,556 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:04,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:11,663 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:11,663 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:22:11,681 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:11,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:11,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:14,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:14,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:14,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:21,697 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:21,697 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:22:21,720 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:21,723 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:21,723 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:24,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:24,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:24,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:31,742 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:31,743 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:22:31,763 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:31,766 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:31,766 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:34,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:34,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:34,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:41,782 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:41,783 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:22:41,808 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:41,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:41,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:44,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:44,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:44,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:51,834 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:51,834 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:22:51,862 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:22:51,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:22:51,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:22:54,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:22:54,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:22:54,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:01,884 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:01,884 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:01,901 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:01,903 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:01,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:04,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:04,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:04,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:11,917 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:11,917 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:11,940 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:11,943 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:11,943 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:14,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:14,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:14,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:21,957 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:21,957 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:21,980 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:21,986 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:21,987 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:24,721 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:24,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:24,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:32,002 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:32,002 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:32,019 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:32,020 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:32,020 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:34,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:34,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:34,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:42,036 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:42,036 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:42,065 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:42,069 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:42,069 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:44,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:44,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:44,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:52,086 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:52,086 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:23:52,108 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:23:52,111 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:23:52,111 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:23:54,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:23:54,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:23:54,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:02,125 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:02,127 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:24:02,148 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:02,151 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:02,151 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:04,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:04,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:04,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:12,165 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:12,165 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:24:12,190 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:12,194 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:12,194 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:14,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:14,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:14,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:22,207 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:22,207 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:24:22,224 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:22,226 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:22,226 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:24,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:24,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:24,869 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:32,242 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:32,242 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:24:32,265 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:32,266 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:32,267 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:34,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:34,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:34,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:42,282 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.51 % 2021-12-16 03:24:42,282 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.51 2021-12-16 03:24:42,302 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:42,305 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:42,305 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:44,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 03:24:44,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:44,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:52,321 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 03:24:52,321 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 36.89 2021-12-16 03:24:52,342 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:24:52,344 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:24:52,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:24:54,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 03:24:54,938 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:24:54,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:02,361 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.65 % 2021-12-16 03:25:02,362 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 37.65 2021-12-16 03:25:02,382 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:02,387 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:02,387 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:04,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.65 % 2021-12-16 03:25:04,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:04,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:12,401 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:12,401 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:25:12,417 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:12,418 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:12,418 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:14,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:14,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:14,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:22,434 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:22,434 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:25:22,469 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:22,471 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:22,471 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:24,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:25,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:25,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:32,484 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:32,484 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:25:32,500 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:32,503 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:32,503 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:35,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:35,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:35,025 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:42,517 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:42,517 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:25:42,542 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:42,542 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:42,543 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:45,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:45,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:45,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:52,558 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:52,558 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:25:52,578 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:25:52,581 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:25:52,581 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:25:55,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:25:55,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:25:55,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:02,595 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:02,595 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:02,611 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:02,614 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:02,614 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:05,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:05,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:05,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:12,628 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:12,628 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:12,647 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:12,650 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:12,650 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:15,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:15,119 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:15,119 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:22,664 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:22,664 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:22,681 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:22,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:22,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:25,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:25,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:25,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:32,697 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:32,697 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:32,717 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:32,719 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:32,719 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:35,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:35,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:35,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:42,729 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:42,730 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:42,747 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:42,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:42,750 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:45,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:45,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:45,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:52,762 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:52,763 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:26:52,780 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:26:52,782 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:26:52,783 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:26:55,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:26:55,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:26:55,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:02,793 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:02,795 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:02,812 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:02,815 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:02,815 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:05,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:05,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:05,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:12,825 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:12,825 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:12,842 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:12,844 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:12,845 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:15,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:15,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:15,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:22,855 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:22,855 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:22,871 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:22,874 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:22,874 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:25,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:25,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:25,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:32,884 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:32,884 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:32,914 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:32,917 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:32,917 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:35,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:35,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:35,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:42,928 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:42,928 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:42,950 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:42,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:42,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:45,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:45,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:45,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:52,967 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:52,967 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.03 2021-12-16 03:27:52,992 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:27:52,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:27:52,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:27:55,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.03 % 2021-12-16 03:27:55,323 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:27:55,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:03,005 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.41 % 2021-12-16 03:28:03,005 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 38.41 2021-12-16 03:28:03,029 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:03,030 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:03,030 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:05,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.41 % 2021-12-16 03:28:05,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:05,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:13,040 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.17 % 2021-12-16 03:28:13,042 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.17 2021-12-16 03:28:13,059 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:13,062 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:13,062 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:15,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.17 % 2021-12-16 03:28:15,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:15,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:23,072 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.17 % 2021-12-16 03:28:23,072 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.17 2021-12-16 03:28:23,092 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:23,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:23,095 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:25,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:25,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:25,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:33,108 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:33,108 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:28:33,125 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:33,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:33,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:35,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:35,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:35,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:43,140 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:43,140 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:28:43,154 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:43,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:43,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:45,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:45,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:45,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:53,170 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:53,170 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:28:53,190 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:28:53,194 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:28:53,194 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:28:55,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:28:55,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:28:55,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:03,207 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:03,207 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:03,224 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:03,227 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:03,227 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:05,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:05,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:05,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:13,259 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:13,259 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:13,280 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:13,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:13,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:15,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:15,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:15,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:23,295 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:23,296 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:23,315 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:23,319 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:23,319 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:25,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:25,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:25,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:33,329 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:33,329 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:33,345 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:33,348 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:33,348 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:35,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:35,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:35,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:43,358 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:43,358 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:43,390 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:43,392 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:43,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:45,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:45,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:45,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:53,403 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:53,404 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:29:53,418 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:29:53,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:29:53,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:29:55,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:29:55,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:29:55,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:03,446 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:03,446 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:03,464 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:03,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:03,467 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:05,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:05,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:05,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:13,477 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:13,477 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:13,496 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:13,499 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:13,500 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:15,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:15,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:15,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:23,509 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:23,509 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:23,525 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:23,526 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:23,526 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:25,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:25,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:25,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:33,538 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:33,538 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:33,555 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:33,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:33,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:35,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:35,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:35,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:43,568 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:43,568 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:43,585 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:43,588 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:43,588 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:45,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:45,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:45,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:53,605 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:53,605 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:30:53,625 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:30:53,627 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:30:53,628 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:30:55,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:30:55,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:30:55,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:03,638 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:31:03,638 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:31:03,657 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:03,660 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:03,660 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:05,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:31:05,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:05,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:13,671 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.55 % 2021-12-16 03:31:13,673 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.55 2021-12-16 03:31:13,690 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:13,694 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:13,694 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:15,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.93 % 2021-12-16 03:31:15,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:15,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:23,704 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.93 % 2021-12-16 03:31:23,704 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 39.93 2021-12-16 03:31:23,721 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:23,723 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:23,723 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:25,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.32 % 2021-12-16 03:31:25,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:25,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:33,734 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.69 % 2021-12-16 03:31:33,736 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 40.69 2021-12-16 03:31:33,752 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:33,755 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:33,755 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:35,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.69 % 2021-12-16 03:31:35,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:35,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:43,765 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.69 % 2021-12-16 03:31:43,765 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 40.69 2021-12-16 03:31:43,782 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:43,783 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:43,785 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:45,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.69 % 2021-12-16 03:31:45,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:45,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:53,796 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:31:53,796 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:31:53,816 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:31:53,819 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:31:53,819 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:31:55,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:31:55,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:31:55,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:03,829 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:03,829 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:03,846 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:03,849 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:03,849 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:05,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:05,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:05,813 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:13,861 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:13,861 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:13,878 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:13,880 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:13,880 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:15,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:15,835 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:15,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:23,891 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:23,891 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:23,907 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:23,910 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:23,910 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:25,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:25,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:25,852 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:33,923 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:33,923 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:33,940 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:33,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:33,944 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:35,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:35,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:35,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:43,954 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:43,956 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:43,973 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:43,974 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:43,974 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:45,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:45,890 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:45,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:53,986 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:53,986 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:32:54,003 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:32:54,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:32:54,006 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:32:55,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:32:55,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:32:55,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:04,016 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:04,016 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:04,038 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:04,039 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:04,039 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:05,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:05,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:05,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:14,051 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:14,051 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:14,071 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:14,072 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:14,073 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:15,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:15,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:15,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:24,082 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:24,084 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:24,098 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:24,101 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:24,101 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:25,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:25,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:25,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:34,111 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:34,111 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:34,131 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:34,132 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:34,132 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:35,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:35,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:35,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:44,142 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:44,144 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:44,164 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:44,167 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:44,167 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:46,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:46,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:46,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:54,177 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:54,177 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:33:54,194 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:33:54,196 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:33:54,196 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:33:56,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:33:56,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:33:56,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:04,206 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:04,207 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:34:04,221 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:04,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:04,223 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:06,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:06,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:06,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:14,243 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:14,243 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:34:14,260 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:14,265 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:14,265 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:16,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:16,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:16,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:24,275 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:24,275 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.07 2021-12-16 03:34:24,290 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:24,292 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:24,292 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:26,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.07 % 2021-12-16 03:34:26,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:26,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:34,305 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.45 % 2021-12-16 03:34:34,305 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 41.45 2021-12-16 03:34:34,325 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:34,328 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:34,328 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:36,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.45 % 2021-12-16 03:34:36,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:36,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:44,341 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.21 % 2021-12-16 03:34:44,341 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.21 2021-12-16 03:34:44,359 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:44,362 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:44,362 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:46,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.21 % 2021-12-16 03:34:46,124 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:46,124 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:54,375 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.21 % 2021-12-16 03:34:54,375 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.21 2021-12-16 03:34:54,392 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:34:54,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:34:54,394 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:34:56,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.21 % 2021-12-16 03:34:56,147 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:34:56,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:04,405 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:04,405 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:04,447 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:04,451 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:04,453 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:06,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:06,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:06,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:14,463 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:14,463 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:14,484 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:14,487 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:14,489 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:16,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:16,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:16,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:24,500 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:24,500 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:24,523 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:24,525 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:24,525 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:26,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:26,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:26,219 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:34,540 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:34,542 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:34,559 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:34,562 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:34,562 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:36,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:36,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:36,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:44,572 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:44,572 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:44,589 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:44,592 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:44,592 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:46,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:46,256 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:46,256 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:54,605 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:54,605 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:35:54,630 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:35:54,632 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:35:54,632 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:35:56,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:35:56,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:35:56,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:04,642 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:04,642 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:04,658 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:04,661 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:04,661 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:06,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:06,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:06,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:14,671 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:14,671 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:14,690 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:14,691 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:14,691 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:16,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:16,313 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:16,313 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:24,707 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:24,707 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:24,727 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:24,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:24,733 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:26,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:26,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:26,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:34,743 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:34,744 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:34,762 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:34,763 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:34,763 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:36,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:36,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:36,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:44,773 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:44,773 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:44,792 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:44,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:44,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:46,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:46,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:46,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:54,808 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:54,808 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:36:54,826 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:36:54,828 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:36:54,829 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:36:56,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:36:56,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:36:56,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:04,841 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:04,841 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:37:04,865 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:04,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:04,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:06,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:06,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:06,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:14,885 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:14,885 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:37:14,921 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:14,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:14,924 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:16,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:16,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:16,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:24,934 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:24,934 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:37:24,951 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:24,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:24,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:26,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:26,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:26,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:34,964 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:34,966 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.59 2021-12-16 03:37:34,983 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:34,984 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:34,984 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:36,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.59 % 2021-12-16 03:37:36,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:36,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:44,996 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.96 % 2021-12-16 03:37:44,996 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 42.96 2021-12-16 03:37:45,015 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:45,019 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:45,019 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:46,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.96 % 2021-12-16 03:37:46,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:46,489 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:55,032 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.72 % 2021-12-16 03:37:55,032 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 43.72 2021-12-16 03:37:55,055 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:37:55,056 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:37:55,056 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:37:56,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.72 % 2021-12-16 03:37:56,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:37:56,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:05,066 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.72 % 2021-12-16 03:38:05,066 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 43.72 2021-12-16 03:38:05,085 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:05,089 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:05,089 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:06,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.72 % 2021-12-16 03:38:06,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:06,528 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:15,101 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:15,101 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:38:15,118 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:15,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:15,121 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:16,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:16,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:16,545 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:25,131 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:25,131 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:38:25,150 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:25,151 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:25,151 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:26,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:26,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:26,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:35,165 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:35,165 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:38:35,186 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:35,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:35,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:36,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:36,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:36,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:45,201 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:45,201 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:38:45,223 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:45,224 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:45,224 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:46,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:46,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:46,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:55,237 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:55,239 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:38:55,260 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:38:55,265 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:38:55,265 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:38:56,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:38:56,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:38:56,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:05,276 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:05,276 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:05,295 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:05,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:05,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:06,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:06,645 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:06,645 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:15,316 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:15,316 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:15,332 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:15,335 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:15,335 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:16,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:16,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:16,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:25,362 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:25,364 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:25,380 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:25,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:25,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:26,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:26,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:26,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:35,394 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:35,394 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:35,410 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:35,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:35,411 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:36,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:36,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:36,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:45,424 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:45,424 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:45,444 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:45,447 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:45,447 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:46,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:46,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:46,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:55,457 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:55,457 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:39:55,473 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:39:55,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:39:55,476 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:39:56,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:39:56,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:39:56,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:05,487 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:05,487 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:05,509 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:05,512 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:05,513 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:06,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:06,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:06,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:15,523 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:15,525 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:15,552 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:15,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:15,556 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:16,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:16,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:16,775 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:25,566 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:25,566 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:25,582 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:25,584 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:25,585 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:26,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:26,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:26,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:35,598 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:35,598 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:35,619 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:35,621 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:35,621 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:36,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:36,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:36,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:45,632 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:45,634 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:45,651 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:45,654 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:45,655 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:46,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:46,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:46,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:55,665 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:55,665 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:40:55,681 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:40:55,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:40:55,684 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:40:56,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:40:56,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:40:56,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:05,694 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:41:05,694 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.11 2021-12-16 03:41:05,711 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:05,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:05,713 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:06,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.11 % 2021-12-16 03:41:06,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:06,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:15,723 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.48 % 2021-12-16 03:41:15,724 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 44.48 2021-12-16 03:41:15,740 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:15,742 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:15,742 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:16,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.48 % 2021-12-16 03:41:16,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:16,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:25,753 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.24 % 2021-12-16 03:41:25,755 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.24 2021-12-16 03:41:25,773 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:25,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:25,776 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:26,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.24 % 2021-12-16 03:41:26,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:26,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:35,788 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.24 % 2021-12-16 03:41:35,789 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.24 2021-12-16 03:41:35,809 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:35,812 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:35,812 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:36,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.24 % 2021-12-16 03:41:36,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:36,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:45,822 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:41:45,822 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:41:45,842 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:45,844 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:45,844 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:46,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:41:46,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:46,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:55,855 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:41:55,855 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:41:55,872 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:41:55,877 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:41:55,877 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:41:56,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:41:56,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:41:56,948 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:05,887 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:05,888 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:05,907 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:05,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:05,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:06,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:06,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:06,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:15,924 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:15,924 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:15,954 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:15,960 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:15,960 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:16,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:16,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:16,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:25,970 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:25,970 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:25,986 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:25,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:25,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:27,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:27,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:27,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:35,999 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:35,999 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:36,013 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:36,015 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:36,016 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:37,020 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:37,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:37,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:46,026 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:46,026 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:46,043 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:46,046 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:46,046 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:47,038 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:47,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:47,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:56,059 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:56,059 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:42:56,076 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:42:56,079 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:42:56,079 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:42:57,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:42:57,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:42:57,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:06,091 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:06,091 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:06,108 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:06,109 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:06,109 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:07,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:07,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:07,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:16,122 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:16,122 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:16,141 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:16,145 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:16,145 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:17,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:17,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:17,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:26,154 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:26,155 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:26,171 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:26,173 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:26,174 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:27,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:27,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:27,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:36,187 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:36,187 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:36,207 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:36,210 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:36,210 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:37,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:37,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:37,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:46,220 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:46,221 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:46,237 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:46,240 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:46,240 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:47,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:47,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:47,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:56,250 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:56,250 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:43:56,266 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:43:56,267 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:43:56,267 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:43:57,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:43:57,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:43:57,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:06,279 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:06,279 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:44:06,296 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:06,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:06,301 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:07,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:07,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:07,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:16,309 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:16,311 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:44:16,328 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:16,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:16,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:17,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:17,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:17,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:26,346 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:26,346 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 45.62 2021-12-16 03:44:26,381 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:26,384 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:26,384 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:27,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.62 % 2021-12-16 03:44:27,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:27,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:36,397 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.38 % 2021-12-16 03:44:36,397 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 46.38 2021-12-16 03:44:36,415 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:36,420 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:36,420 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:37,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.38 % 2021-12-16 03:44:37,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:37,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:46,431 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.76 % 2021-12-16 03:44:46,433 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 46.76 2021-12-16 03:44:46,448 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:46,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:46,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:47,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.76 % 2021-12-16 03:44:47,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:47,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:56,463 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.76 % 2021-12-16 03:44:56,463 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 46.76 2021-12-16 03:44:56,480 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:44:56,480 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:44:56,482 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:44:57,278 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:44:57,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:44:57,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:06,493 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:06,493 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:06,513 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:06,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:06,516 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:07,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:07,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:07,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:16,530 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:16,530 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:16,549 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:16,552 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:16,552 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:17,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:17,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:17,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:26,568 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:26,568 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:26,592 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:26,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:26,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:27,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:27,349 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:27,349 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:36,607 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:36,607 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:36,630 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:36,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:36,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:37,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:37,367 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:37,367 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:46,651 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:46,653 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:46,667 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:46,671 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:46,671 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:47,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:47,388 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:47,388 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:56,681 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:56,681 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:45:56,703 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:45:56,704 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:45:56,704 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:45:57,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:45:57,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:45:57,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:06,714 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:06,714 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:06,733 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:06,736 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:06,736 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:07,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:07,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:07,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:16,746 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:16,746 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:16,772 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:16,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:16,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:17,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:17,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:17,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:26,786 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:26,786 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:26,805 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:26,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:26,809 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:27,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:27,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:27,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:36,819 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:36,819 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:36,835 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:36,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:36,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:37,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:37,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:37,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:46,848 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:46,849 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:46,867 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:46,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:46,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:47,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:47,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:47,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:56,880 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:56,880 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:46:56,897 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:46:56,900 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:46:56,900 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:46:57,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:46:57,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:46:57,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:06,910 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:06,910 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:47:06,927 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:06,930 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:06,930 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:07,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:07,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:07,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:16,940 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:16,940 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:47:16,963 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:16,966 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:16,966 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:17,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:17,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:17,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:26,977 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:26,979 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:47:26,994 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:26,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:26,996 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:27,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:27,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:27,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:37,007 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:37,007 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 47.13 2021-12-16 03:47:37,025 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:37,026 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:37,026 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:37,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.13 % 2021-12-16 03:47:37,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:37,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:47,039 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.27 % 2021-12-16 03:47:47,039 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.27 2021-12-16 03:47:47,059 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:47,063 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:47,063 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:47,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.27 % 2021-12-16 03:47:47,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:47,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:57,073 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:47:57,075 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:47:57,089 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:47:57,092 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:47:57,092 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:47:57,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:47:57,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:47:57,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:07,104 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:07,104 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:07,122 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:07,125 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:07,125 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:07,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:07,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:07,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:17,137 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:17,137 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:17,154 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:17,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:17,158 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:17,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:17,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:17,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:27,168 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:27,170 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:27,187 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:27,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:27,190 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:27,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:27,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:27,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:37,213 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:37,213 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:37,230 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:37,234 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:37,234 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:37,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:37,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:37,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:47,243 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:47,244 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:47,260 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:47,263 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:47,263 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:47,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:47,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:47,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:57,273 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:57,273 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:48:57,295 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:48:57,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:48:57,296 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:48:57,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:48:57,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:48:57,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:07,306 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:07,306 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:07,325 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:07,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:07,329 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:07,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:07,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:07,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:17,342 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:17,342 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:17,358 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:17,359 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:17,361 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:17,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:17,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:17,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:27,369 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:27,371 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:27,387 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:27,388 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:27,388 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:27,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:27,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:27,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:37,400 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:37,400 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:37,420 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:37,421 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:37,423 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:37,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:37,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:37,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:47,433 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:47,434 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:47,453 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:47,454 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:47,456 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:47,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:47,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:47,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:57,469 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:57,469 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:49:57,487 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:49:57,490 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:49:57,490 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:49:57,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:49:57,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:49:57,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:07,502 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:07,502 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:07,517 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:07,519 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:07,519 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:07,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:07,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:07,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:17,532 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:17,533 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:17,555 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:17,559 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:17,559 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:17,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:17,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:17,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:27,569 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:27,569 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:27,586 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:27,588 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:27,588 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:27,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:27,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:27,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:37,599 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:37,599 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:37,621 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:37,624 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:37,624 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:37,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:37,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:37,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:47,634 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:47,634 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:47,654 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:47,655 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:47,657 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:47,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:47,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:47,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:57,667 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:57,667 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:50:57,690 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:50:57,694 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:50:57,694 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:50:58,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:50:58,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:50:58,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:07,707 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:07,707 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:51:07,724 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:07,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:07,726 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:08,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:08,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:08,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:17,740 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:17,740 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:51:17,763 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:17,766 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:17,766 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:18,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:18,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:18,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:27,778 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:27,778 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 48.65 2021-12-16 03:51:27,799 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:27,802 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:27,802 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:28,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.65 % 2021-12-16 03:51:28,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:28,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:37,813 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:37,815 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:51:37,832 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:37,835 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:37,835 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:38,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:38,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:38,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:47,846 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:47,846 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:51:47,865 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:47,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:47,868 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:48,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:48,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:48,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:57,878 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:57,878 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:51:57,892 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:51:57,894 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:51:57,894 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:51:58,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:51:58,135 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:51:58,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:07,907 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:07,907 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:07,931 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:07,934 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:07,934 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:08,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:08,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:08,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:17,944 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:17,944 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:17,960 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:17,961 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:17,961 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:18,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:18,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:18,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:27,971 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:27,971 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:27,990 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:27,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:27,993 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:28,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:28,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:28,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:38,003 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:38,003 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:38,020 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:38,025 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:38,025 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:38,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:38,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:38,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:48,039 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:48,039 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:48,058 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:48,061 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:48,061 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:48,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:48,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:48,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:58,072 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:58,072 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:52:58,088 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:52:58,091 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:52:58,091 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:52:58,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:52:58,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:52:58,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:08,101 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:08,102 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:08,125 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:08,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:08,128 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:08,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:08,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:08,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:18,138 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:18,138 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:18,155 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:18,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:18,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:18,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:18,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:18,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:28,167 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:28,167 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:28,184 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:28,186 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:28,186 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:28,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:28,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:28,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:38,197 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:38,198 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:38,214 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:38,217 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:38,217 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:38,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:38,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:38,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:48,229 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:48,230 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:48,246 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:48,249 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:48,249 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:48,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:48,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:48,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:58,259 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:58,259 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:53:58,280 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:53:58,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:53:58,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:53:58,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:53:58,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:53:58,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:08,296 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:08,296 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:08,313 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:08,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:08,315 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:08,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:08,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:08,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:18,325 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:18,325 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:18,342 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:18,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:18,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:18,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:18,404 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:18,404 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:28,358 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:28,358 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:28,378 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:28,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:28,381 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:28,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:28,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:28,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:38,394 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:38,394 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:38,411 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:38,414 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:38,414 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:38,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:38,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:38,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:48,447 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:48,447 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:48,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:48,466 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:48,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:48,471 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:48,474 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:48,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:58,483 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:58,483 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 03:54:58,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 03:54:58,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:54:58,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:54:58,509 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:54:58,510 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:54:58,512 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:08,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.31 % 2021-12-16 03:55:08,512 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:08,512 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:08,523 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.31 % 2021-12-16 03:55:08,523 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.31 2021-12-16 03:55:08,542 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:08,545 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:08,545 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:18,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:18,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:18,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:18,555 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:18,555 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:55:18,576 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:18,579 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:18,579 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:28,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:28,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:28,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:28,592 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:28,592 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:55:28,611 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:28,612 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:28,612 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:38,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:38,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:38,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:38,628 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:38,628 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:55:38,651 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:38,654 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:38,654 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:48,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:48,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:48,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:48,667 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:48,667 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:55:48,687 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:48,690 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:48,690 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:58,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:58,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:55:58,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:55:58,703 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:55:58,703 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:55:58,720 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:55:58,721 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:55:58,721 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:08,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:08,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:08,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:08,736 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:08,736 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:08,769 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:08,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:08,773 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:18,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:18,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:18,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:18,783 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:18,783 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:18,802 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:18,806 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:18,806 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:28,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:28,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:28,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:28,819 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:28,819 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:28,838 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:28,842 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:28,842 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:38,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:38,681 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:38,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:38,854 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:38,854 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:38,875 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:38,877 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:38,877 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:48,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:48,701 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:48,701 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:48,892 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:48,894 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:48,918 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:48,923 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:48,923 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:58,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:58,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:56:58,737 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:56:58,934 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:56:58,934 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:56:58,951 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:56:58,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:56:58,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:08,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:08,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:08,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:08,963 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:08,964 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:08,986 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:08,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:08,989 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:18,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:18,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:18,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:18,999 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:19,000 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:19,019 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:19,022 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:19,023 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:28,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:28,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:28,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:29,035 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:29,035 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:29,056 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:29,059 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:29,059 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:38,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:38,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:38,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:39,085 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:39,086 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:39,107 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:39,109 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:39,111 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:48,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:48,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:48,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:49,121 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:49,121 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:49,138 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:49,140 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:49,140 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:58,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:58,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:57:58,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:57:59,151 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:57:59,151 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:57:59,167 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:57:59,168 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:57:59,168 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:08,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:58:08,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:08,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:09,187 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:58:09,188 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:58:09,207 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:09,209 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:09,209 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:18,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:58:18,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:18,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:19,220 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.69 % 2021-12-16 03:58:19,220 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 51.69 2021-12-16 03:58:19,236 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:19,237 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:19,237 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:28,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.07 % 2021-12-16 03:58:28,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:28,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:29,250 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.07 % 2021-12-16 03:58:29,250 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 52.07 2021-12-16 03:58:29,263 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:29,266 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:29,266 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:38,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.83 % 2021-12-16 03:58:38,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:38,915 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:39,276 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.83 % 2021-12-16 03:58:39,276 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 52.83 2021-12-16 03:58:39,296 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:39,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:39,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:48,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:58:48,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:48,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:49,312 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:58:49,312 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:58:49,331 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:49,332 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:49,332 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:58,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:58:58,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:58:58,951 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:58:59,345 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:58:59,345 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:58:59,365 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:58:59,368 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:58:59,368 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:08,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:08,967 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:08,967 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:09,378 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:09,378 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:09,397 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:09,400 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:09,400 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:18,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:18,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:18,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:19,411 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:19,413 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:19,433 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:19,434 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:19,434 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:29,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:29,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:29,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:29,447 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:29,448 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:29,470 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:29,473 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:29,473 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:39,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:39,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:39,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:39,483 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:39,483 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:39,500 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:39,503 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:39,503 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:49,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:49,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:49,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:49,516 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:49,516 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:49,535 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:49,536 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:49,536 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:59,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:59,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 03:59:59,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 03:59:59,549 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 03:59:59,551 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 03:59:59,569 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 03:59:59,571 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 03:59:59,572 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:09,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:09,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:09,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:09,582 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:09,582 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:09,598 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:09,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:09,601 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:19,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:19,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:19,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:19,611 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:19,611 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:19,628 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:19,630 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:19,630 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:29,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:29,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:29,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:29,641 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:29,641 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:29,658 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:29,661 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:29,661 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:39,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:39,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:39,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:39,673 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:39,674 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:39,690 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:39,693 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:39,693 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:49,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:49,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:49,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:49,703 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:49,703 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:49,723 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:49,724 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:49,724 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:59,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:59,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:00:59,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:00:59,736 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:00:59,736 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:00:59,753 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:00:59,755 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:00:59,756 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:09,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:09,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:09,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:09,770 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:09,770 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:01:09,789 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:09,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:09,792 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:19,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:19,220 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:19,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:19,805 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:19,805 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:01:19,825 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:19,828 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:19,828 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:29,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:29,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:29,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:29,839 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:29,839 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:01:29,861 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:29,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:29,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:39,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:39,252 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:39,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:39,875 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.21 % 2021-12-16 04:01:39,877 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.21 2021-12-16 04:01:39,907 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:39,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:39,908 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:49,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.59 % 2021-12-16 04:01:49,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:49,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:49,920 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.59 % 2021-12-16 04:01:49,920 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.59 2021-12-16 04:01:49,937 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:49,940 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:49,940 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:59,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.59 % 2021-12-16 04:01:59,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:01:59,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:01:59,960 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.59 % 2021-12-16 04:01:59,960 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 53.59 2021-12-16 04:01:59,979 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:01:59,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:01:59,983 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:09,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.97 % 2021-12-16 04:02:09,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:09,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:09,993 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.35 % 2021-12-16 04:02:09,994 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.35 2021-12-16 04:02:10,009 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:02:10,012 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:02:10,012 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:19,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.35 % 2021-12-16 04:02:19,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:19,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:20,023 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:20,023 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:02:20,046 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:02:20,049 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:02:20,049 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:29,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:29,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:29,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:30,061 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:30,061 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:02:30,081 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:02:30,085 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:02:30,085 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:39,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:39,364 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:39,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:40,095 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:40,095 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:02:40,114 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:02:40,115 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:02:40,115 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:49,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:49,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:49,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:50,131 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:50,131 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:02:50,154 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:02:50,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:02:50,157 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:02:59,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:02:59,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:02:59,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:00,167 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:00,167 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:00,184 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:00,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:00,187 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:09,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:09,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:09,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:10,197 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:10,197 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:10,217 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:10,219 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:10,219 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:19,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:19,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:19,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:20,233 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:20,233 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:20,259 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:20,263 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:20,263 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:29,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:29,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:29,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:30,275 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:30,275 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:30,298 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:30,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:30,299 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:39,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:39,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:39,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:40,309 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:40,311 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:40,325 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:40,328 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:40,328 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:49,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:49,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:49,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:50,342 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:50,342 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:03:50,364 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:03:50,367 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:03:50,367 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:03:59,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:03:59,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:03:59,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:00,384 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:00,384 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:00,403 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:00,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:00,407 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:09,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:09,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:09,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:10,417 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:10,417 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:10,436 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:10,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:10,437 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:19,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:19,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:19,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:20,450 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:20,450 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:20,469 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:20,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:20,470 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:29,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:29,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:29,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:30,480 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:30,480 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:30,496 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:30,497 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:30,497 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:39,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:39,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:39,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:40,509 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:40,509 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:40,528 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:40,530 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:40,530 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:49,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:49,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:49,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:50,542 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:50,542 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:04:50,559 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:04:50,561 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:04:50,562 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:04:59,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:04:59,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:04:59,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:00,572 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:05:00,572 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:05:00,594 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:00,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:00,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:09,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:05:09,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:09,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:10,614 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.73 % 2021-12-16 04:05:10,614 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 54.73 2021-12-16 04:05:10,637 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:10,640 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:10,641 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:19,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.10 % 2021-12-16 04:05:19,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:19,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:20,654 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.10 % 2021-12-16 04:05:20,654 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 55.1 2021-12-16 04:05:20,680 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:20,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:20,681 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:29,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.10 % 2021-12-16 04:05:29,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:29,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:30,694 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.10 % 2021-12-16 04:05:30,694 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 55.1 2021-12-16 04:05:30,717 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:30,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:30,720 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:39,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.86 % 2021-12-16 04:05:39,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:39,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:40,732 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.86 % 2021-12-16 04:05:40,732 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 55.86 2021-12-16 04:05:40,763 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:40,765 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:40,765 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:49,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:05:49,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:49,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:50,776 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:05:50,776 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:05:50,802 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:05:50,803 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:05:50,803 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:05:59,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:05:59,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:05:59,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:00,815 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:00,815 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:00,834 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:00,836 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:00,838 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:09,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:09,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:09,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:10,846 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:10,846 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:10,862 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:10,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:10,865 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:19,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:19,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:19,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:20,878 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:20,878 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:20,901 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:20,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:20,904 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:29,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:29,819 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:29,819 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:30,914 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:30,914 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:30,934 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:30,937 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:30,937 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:39,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:39,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:39,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:40,947 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:40,947 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:40,967 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:40,969 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:40,969 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:49,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:49,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:49,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:50,979 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:50,979 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:06:50,996 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:06:50,999 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:06:50,999 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:06:59,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:06:59,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:06:59,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:01,009 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:01,009 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:01,023 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:01,026 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:01,026 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:09,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:09,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:09,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:11,040 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:11,040 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:11,065 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:11,066 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:11,066 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:19,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:19,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:19,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:21,092 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:21,092 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:21,112 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:21,115 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:21,115 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:29,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:29,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:29,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:31,125 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:31,125 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:31,141 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:31,144 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:31,144 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:39,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:39,951 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:39,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:41,157 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:41,157 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:41,180 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:41,181 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:41,183 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:49,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:49,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:49,977 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:51,194 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:51,194 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:07:51,211 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:07:51,214 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:07:51,214 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:07:59,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:07:59,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:07:59,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:01,229 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:01,229 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:08:01,247 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:01,250 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:01,250 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:10,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:10,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:08:10,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:11,262 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:11,263 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:08:11,282 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:11,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:11,285 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:20,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:20,030 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:08:20,030 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:21,296 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:21,296 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:08:21,315 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:21,319 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:21,319 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:30,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:30,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:08:30,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:31,332 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 04:08:31,332 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.23 2021-12-16 04:08:31,351 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:31,354 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:31,354 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:40,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.61 % 2021-12-16 04:08:40,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:08:40,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:41,368 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.61 % 2021-12-16 04:08:41,368 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.61 2021-12-16 04:08:41,385 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:41,388 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:41,388 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:50,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.61 % 2021-12-16 04:08:50,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:08:50,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:08:51,398 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.61 % 2021-12-16 04:08:51,398 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 56.61 2021-12-16 04:08:51,415 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:08:51,417 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:08:51,418 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:00,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.99 % 2021-12-16 04:09:00,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:00,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:01,430 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.36 % 2021-12-16 04:09:01,430 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.36 2021-12-16 04:09:01,447 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:01,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:01,450 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:10,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:10,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:10,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:11,463 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:11,463 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:09:11,493 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:11,496 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:11,496 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:20,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:20,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:20,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:21,507 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:21,507 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:09:21,526 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:21,529 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:21,529 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:30,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:30,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:30,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:31,539 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:31,539 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:09:31,556 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:31,559 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:31,559 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:40,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:40,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:40,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:41,572 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:41,572 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:09:41,592 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:41,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:41,595 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:50,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:50,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:09:50,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:09:51,607 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:09:51,607 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:09:51,634 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:09:51,637 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:09:51,637 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:00,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:00,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:00,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:01,648 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:01,648 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:01,667 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:01,670 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:01,670 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:10,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:10,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:10,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:11,680 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:11,680 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:11,697 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:11,700 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:11,700 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:20,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:20,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:20,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:21,713 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:21,713 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:21,733 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:21,736 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:21,736 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:30,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:30,275 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:30,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:31,746 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:31,746 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:31,765 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:31,767 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:31,767 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:40,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:40,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:40,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:41,778 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:41,778 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:41,793 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:41,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:41,796 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:50,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:50,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:10:50,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:10:51,809 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:10:51,809 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:10:51,826 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:10:51,829 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:10:51,831 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:00,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:00,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:00,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:01,839 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:01,841 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:01,855 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:01,858 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:01,858 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:10,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:10,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:10,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:11,871 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:11,871 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:11,890 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:11,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:11,891 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:20,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:20,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:20,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:21,901 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:21,903 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:21,918 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:21,921 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:21,921 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:30,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:30,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:30,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:31,931 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:31,933 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:31,950 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:31,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:31,953 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:40,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:40,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:40,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:41,963 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:41,963 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:41,984 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:41,986 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:41,987 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:50,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:50,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:11:50,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:11:51,996 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.74 % 2021-12-16 04:11:51,997 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 57.74 2021-12-16 04:11:52,013 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:11:52,016 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:11:52,016 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:00,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.12 % 2021-12-16 04:12:00,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:00,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:02,028 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.12 % 2021-12-16 04:12:02,028 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 58.12 2021-12-16 04:12:02,042 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:02,045 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:02,046 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:10,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.12 % 2021-12-16 04:12:10,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:10,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:12,056 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.12 % 2021-12-16 04:12:12,058 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 58.12 2021-12-16 04:12:12,078 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:12,082 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:12,082 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:20,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.49 % 2021-12-16 04:12:20,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:20,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:22,092 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.49 % 2021-12-16 04:12:22,092 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 58.49 2021-12-16 04:12:22,108 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:22,108 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:22,109 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:30,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:30,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:30,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:32,121 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:32,122 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:12:32,147 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:32,148 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:32,148 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:40,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:40,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:40,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:42,158 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:42,158 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:12:42,177 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:42,180 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:42,180 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:50,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:50,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:12:50,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:12:52,190 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:12:52,191 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:12:52,209 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:12:52,213 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:12:52,213 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:00,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:00,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:00,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:02,223 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:02,224 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:13:02,243 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:02,247 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:02,247 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:10,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:10,595 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:10,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:12,259 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:12,259 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:13:12,279 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:12,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:12,282 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:20,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:20,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:20,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:22,292 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:22,292 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:13:22,309 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:22,311 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:22,311 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:30,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:30,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:30,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:32,325 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:32,325 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:13:32,342 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:32,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:32,345 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:40,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:40,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:40,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:42,355 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.25 % 2021-12-16 04:13:42,355 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 59.25 2021-12-16 04:13:42,380 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:42,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:42,382 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:50,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.62 % 2021-12-16 04:13:50,678 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:13:50,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:13:52,395 | test | DEBUG | pool-3-thread-3 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.00 % 2021-12-16 04:13:52,395 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: running, progress: 60.0 2021-12-16 04:13:52,421 | infra | DEBUG | pool-3-thread-3 | [task:check:412] Wait before next rebalance progress check 2021-12-16 04:13:52,430 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 04:13:52,430 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:14:00,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.00 % 2021-12-16 04:14:00,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:14:00,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:14:02,443 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 04:14:02,443 | infra | DEBUG | pool-3-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 04:14:07,453 | test | INFO | pool-3-thread-3 | [task:check:385] Rebalance - status: none, progress: 100 2021-12-16 04:14:07,594 | test | INFO | pool-3-thread-3 | [task:check:444] Rebalance completed with progress: 100% in 7852.25200009 sec 2021-12-16 04:14:07,594 | infra | INFO | pool-3-thread-3 | [task:complete_task:105] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639648995.22' completed 2021-12-16 04:14:10,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 04:14:10,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 04:14:15,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 04:14:15,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 04:14:25,740 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >100% in 7750.46499991 seconds 2021-12-16 04:14:25,750 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2021-12-16 04:14:25,750 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.66]_1639648995.22 2021-12-16 04:14:25,789 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 46516.5, u'diskFetches': 37232687, u'dataUsed': 9402951293804L, u'itemCount': 4361437693L, u'quotaPercentUsed': 77.18447826430202, u'storageTotals': {u'hdd': {u'usedByData': 11670557024665L, u'used': 11903089833405L, u'quotaTotal': 47967746654208L, u'total': 47967746654208L, u'free': 34106304259692L}, u'ram': {u'usedByData': 331504809904L, u'used': 774199152640L, u'quotaTotal': 776848015360L, u'total': 1079482892288L, u'quotaUsed': 429496729600L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 11670557024665L, u'vbActiveNumNonResident': 4196689282L, u'memUsed': 331504809904L} 2021-12-16 04:14:26,006 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4361437693 | 400.00 GiB | 308.74 GiB | 10.61 TiB | 3.77738770095 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-16 04:14:26,006 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-16 04:14:26,288 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.77738770095 2021-12-16 04:14:26,289 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:14:26,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:14:26,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:14:26,755 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:14:26,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:14:26,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:14:26,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:14:26,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:14:26,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:14:26,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:14:26,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:14:26,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 04:14:27,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 04:14:27,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 04:14:27,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 04:14:27,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 04:14:28,747 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:14:28,750 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:14:28,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:14:29,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:14:29,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:14:29,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:14:29,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:14:29,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:14:29,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:14:29,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:14:29,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:14:29,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:14:29,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 04:14:29,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 04:14:29,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 04:14:29,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 04:14:30,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 04:14:32,566 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 305 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 321 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 314 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 301 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 302 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 330 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 341 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 321 2021-12-16 04:14:32,568 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 313 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 313 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-802 is 318 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 311 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 304 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 324 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 318 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 313 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 339 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 331 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 327 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 318 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 349 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 315 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 333 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 310 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 309 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 342 2021-12-16 04:14:32,569 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 310 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 317 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-447 is 335 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 315 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 306 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 308 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 309 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-809 is 302 2021-12-16 04:14:32,571 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 317 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 309 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 306 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 345 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 316 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 326 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 322 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 329 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 308 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 327 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 345 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-794 is 320 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 303 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 339 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-74 is 326 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 315 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 304 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 332 2021-12-16 04:14:32,572 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 301 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 319 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 331 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-381 is 301 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 314 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 309 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 337 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-454 is 311 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 304 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 311 2021-12-16 04:14:32,573 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 317 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-815 is 307 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 323 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 323 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 317 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 310 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 321 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-817 is 317 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-457 is 315 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 321 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 300 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 300 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 311 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 344 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 325 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 311 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 340 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-821 is 300 2021-12-16 04:14:32,575 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 307 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 312 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 324 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 309 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 301 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 320 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 328 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 313 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 308 2021-12-16 04:14:32,576 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 319 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 310 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 302 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-823 is 301 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 321 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 317 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 312 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 327 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 337 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 311 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 337 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 329 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 316 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 314 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 340 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 314 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 322 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 329 2021-12-16 04:14:32,578 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 306 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-827 is 305 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 310 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 307 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 314 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 304 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 321 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 306 2021-12-16 04:14:32,579 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 317 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 337 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 324 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-469 is 302 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 344 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 345 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-830 is 331 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 309 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 303 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 327 2021-12-16 04:14:32,581 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 331 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 318 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-38 is 330 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 300 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 316 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-255 is 300 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 310 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 323 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 329 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 316 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 303 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 300 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 327 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-760 is 300 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 312 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 303 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 318 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 324 2021-12-16 04:14:32,582 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 332 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 341 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 322 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 318 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 315 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 320 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-835 is 302 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 313 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 338 2021-12-16 04:14:32,584 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 300 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 337 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 306 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 312 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 324 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 311 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 314 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 331 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 312 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 308 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 312 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 334 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 326 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 305 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-191 is 317 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 302 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 313 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 322 2021-12-16 04:14:32,585 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-481 is 344 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 303 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 324 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 321 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 322 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 305 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 311 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 312 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 335 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 301 2021-12-16 04:14:32,586 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 306 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 333 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 309 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 310 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-410 is 309 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 338 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 323 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-771 is 316 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-699 is 311 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 328 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 321 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-51 is 333 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 343 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 310 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 301 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 324 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 321 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 316 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 304 2021-12-16 04:14:32,588 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 329 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 328 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 308 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 323 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 307 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 305 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 308 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 305 2021-12-16 04:14:32,589 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 332 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 303 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 353 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 304 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 303 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 309 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 338 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 319 2021-12-16 04:14:32,591 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 317 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 337 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 316 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 323 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 322 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 306 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 305 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-491 is 327 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 311 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 303 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 325 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 318 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 333 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 322 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 313 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-6 is 323 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 312 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 327 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 319 2021-12-16 04:14:32,592 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 312 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 325 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 307 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 302 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 300 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-61 is 308 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 309 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 307 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 320 2021-12-16 04:14:32,594 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 336 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 306 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 321 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 322 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 315 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-64 is 307 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 348 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 332 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 309 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 327 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 326 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 310 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 304 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 306 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 314 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 319 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 317 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 318 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 324 2021-12-16 04:14:32,595 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 320 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 327 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 332 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 333 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 314 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 336 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 309 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-79 is 310 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 323 2021-12-16 04:14:32,596 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 308 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 357 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-358 is 302 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 315 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 336 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 300 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 356 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 314 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 316 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 310 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 324 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 305 2021-12-16 04:14:32,598 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 330 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 306 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 319 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-801 is 304 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 318 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 312 2021-12-16 04:14:32,599 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 343 2021-12-16 04:14:32,601 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 330 2021-12-16 04:14:32,601 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:14:32,604 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:14:32,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:14:33,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:14:33,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:14:33,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:14:33,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:14:33,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:14:33,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:14:33,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:14:33,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:14:33,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:14:33,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 04:14:33,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 04:14:33,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 04:14:33,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 04:14:33,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 04:14:36,121 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-1008 is 326 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-864 is 316 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-144 is 331 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-72 is 303 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 325 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-145 is 313 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 312 2021-12-16 04:14:36,122 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-514 is 334 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-875 is 318 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 339 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 313 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-371 is 327 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 322 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-949 is 320 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-517 is 328 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-157 is 335 2021-12-16 04:14:36,124 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 320 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 315 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-87 is 336 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-88 is 304 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-593 is 321 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-377 is 301 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-89 is 309 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-595 is 318 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-523 is 325 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 312 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-91 is 311 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-578 is 336 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-92 is 312 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-885 is 311 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-597 is 341 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-93 is 327 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-526 is 320 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-959 is 311 2021-12-16 04:14:36,125 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-527 is 309 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-167 is 302 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-95 is 312 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-960 is 313 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-888 is 319 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-528 is 311 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-384 is 320 2021-12-16 04:14:36,127 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-385 is 317 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-98 is 315 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-387 is 307 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-388 is 300 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-100 is 320 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-893 is 324 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-101 is 327 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-579 is 306 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-363 is 318 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-966 is 320 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-534 is 311 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-391 is 307 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-103 is 311 2021-12-16 04:14:36,128 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-536 is 311 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-537 is 323 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 313 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-106 is 304 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-107 is 310 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-540 is 321 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-396 is 322 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-397 is 316 2021-12-16 04:14:36,130 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-974 is 302 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-903 is 306 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-1012 is 310 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-76 is 311 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-904 is 303 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-544 is 335 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-401 is 309 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-547 is 319 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 304 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 315 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-981 is 306 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-549 is 300 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 316 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-551 is 312 2021-12-16 04:14:36,131 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 325 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 316 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-985 is 324 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-1013 is 330 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-941 is 314 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-581 is 301 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-149 is 340 2021-12-16 04:14:36,132 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-915 is 332 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-555 is 324 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-123 is 304 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 333 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-989 is 306 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-413 is 319 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-342 is 311 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 306 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-343 is 320 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-993 is 323 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 340 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-418 is 305 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-346 is 313 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-130 is 318 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-563 is 311 2021-12-16 04:14:36,134 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-347 is 320 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-582 is 326 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-366 is 320 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-348 is 335 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-132 is 329 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-565 is 332 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-133 is 305 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-998 is 321 2021-12-16 04:14:36,135 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-566 is 301 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-350 is 337 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-134 is 312 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-351 is 315 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-135 is 321 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-857 is 317 2021-12-16 04:14:36,137 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-353 is 318 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-137 is 312 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-1002 is 308 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-426 is 315 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-354 is 319 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-138 is 323 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-1003 is 314 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-571 is 311 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-427 is 315 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-355 is 321 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-357 is 302 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-1015 is 308 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-367 is 349 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-574 is 336 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-142 is 326 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-1007 is 319 2021-12-16 04:14:36,138 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-944 is 304 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-584 is 316 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-368 is 306 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 327 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-1017 is 312 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-585 is 324 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-513 is 308 2021-12-16 04:14:36,140 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-81 is 325 2021-12-16 04:14:36,140 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:14:36,144 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:14:36,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:14:36,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:14:36,546 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:14:36,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:14:36,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:14:36,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:14:36,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:14:36,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:14:36,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:14:36,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:14:36,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 04:14:36,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 04:14:36,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 04:14:37,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 04:14:37,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 04:14:38,598 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:14:38,598 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 11182892MB 2021-12-16 04:14:38,598 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 10356MB 2021-12-16 04:14:38,599 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 388026MB 2021-12-16 04:14:38,599 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 10794475MB 2021-12-16 04:14:38,601 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:14:38,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:14:38,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:14:38,996 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:14:38,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:14:39,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:14:39,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:14:39,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:14:39,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:14:39,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:14:39,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:14:39,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 04:14:39,165 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-16 04:14:39,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 04:14:39,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 04:14:39,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 04:14:40,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 04:14:40,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 04:14:41,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 04:14:41,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 04:14:41,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 04:14:42,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 04:14:42,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 04:14:42,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 04:14:43,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 04:14:43,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 04:14:44,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 04:14:44,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 04:14:44,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 04:14:45,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 04:14:45,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 04:14:46,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 04:14:46,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 04:14:46,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 04:14:47,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 04:14:47,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 04:14:47,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 04:14:48,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 04:14:48,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 04:14:48,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 04:14:49,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 04:14:49,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 04:14:50,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 04:14:50,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 04:14:50,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 04:14:51,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 04:14:51,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 04:14:51,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 04:14:52,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 04:14:52,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 04:14:52,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 04:14:53,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 04:14:53,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 04:14:53,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 04:14:54,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 04:14:54,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 04:14:55,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 04:14:55,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 04:14:55,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 04:14:56,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 04:14:56,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 04:14:56,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 04:14:57,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 04:14:57,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 04:14:57,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 04:14:58,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 04:14:58,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 04:14:59,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 04:14:59,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 04:14:59,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 04:15:00,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 04:15:00,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 04:15:00,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 04:15:01,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 04:15:01,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 04:15:01,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 04:15:02,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 04:15:02,505 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:15:02,507 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:15:02,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:15:02,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:15:02,963 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:15:02,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:15:02,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:15:02,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:15:03,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:15:03,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:15:03,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:15:03,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:15:03,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 04:15:03,111 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.68 - core(s): 72 2021-12-16 04:15:03,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 04:15:03,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 04:15:03,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 04:15:04,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 04:15:04,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 04:15:04,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 04:15:05,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 04:15:05,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 04:15:05,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 04:15:06,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 04:15:06,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 04:15:07,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 04:15:07,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 04:15:07,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 04:15:08,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 04:15:08,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 04:15:08,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 04:15:09,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 04:15:09,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 04:15:10,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 04:15:10,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 04:15:10,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 04:15:11,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 04:15:11,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 04:15:11,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 04:15:12,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 04:15:12,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 04:15:13,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 04:15:13,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 04:15:13,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 04:15:14,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 04:15:14,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 04:15:14,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 04:15:15,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 04:15:15,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 04:15:15,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 04:15:16,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 04:15:16,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 04:15:17,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 04:15:17,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 04:15:17,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 04:15:18,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 04:15:18,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 04:15:18,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 04:15:19,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 04:15:19,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 04:15:19,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 04:15:20,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 04:15:20,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 04:15:21,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 04:15:21,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 04:15:21,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 04:15:22,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 04:15:22,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 04:15:23,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 04:15:23,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 04:15:23,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 04:15:24,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 04:15:24,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 04:15:24,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 04:15:25,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 04:15:25,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 04:15:25,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 04:15:26,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 04:15:26,723 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:15:26,726 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:15:26,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:15:27,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:15:27,130 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:15:27,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:15:27,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:15:27,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:15:27,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:15:27,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:15:27,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:15:27,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:15:27,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 04:15:27,348 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.67 - core(s): 72 2021-12-16 04:15:27,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 04:15:27,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 04:15:28,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 04:15:28,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 04:15:28,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 04:15:29,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 04:15:29,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 04:15:30,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 04:15:30,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 04:15:30,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 04:15:31,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 04:15:31,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 04:15:31,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 04:15:32,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 04:15:32,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 04:15:32,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 04:15:33,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 04:15:33,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 04:15:34,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 04:15:34,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 04:15:34,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 04:15:35,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 04:15:35,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 04:15:35,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 04:15:36,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 04:15:36,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 04:15:37,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 04:15:37,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 04:15:37,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 04:15:38,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 04:15:38,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 04:15:39,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 04:15:39,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 04:15:39,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 04:15:40,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 04:15:40,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 04:15:40,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 04:15:41,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 04:15:41,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 04:15:42,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 04:15:42,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 04:15:42,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 04:15:43,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 04:15:43,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 04:15:44,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 04:15:44,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 04:15:44,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 04:15:45,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 04:15:45,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 04:15:46,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 04:15:46,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 04:15:46,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 04:15:47,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 04:15:47,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 04:15:48,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 04:15:48,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 04:15:48,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 04:15:49,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 04:15:49,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 04:15:50,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 04:15:50,362 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 04:15:50,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 04:15:51,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 04:15:51,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 04:15:51,961 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:15:51,963 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:15:52,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:15:52,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:15:52,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:15:52,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:15:52,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:15:52,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:15:52,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:15:52,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:15:52,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:15:52,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:15:52,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 04:15:52,538 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-16 04:15:52,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 04:15:52,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 04:15:53,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 04:15:53,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 04:15:54,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 04:15:54,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 04:15:54,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 04:15:55,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 04:15:55,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 04:15:55,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 04:15:56,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 04:15:56,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 04:15:57,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 04:15:57,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 04:15:57,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 04:15:58,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 04:15:58,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 04:15:58,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 04:15:59,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 04:15:59,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 04:16:00,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 04:16:00,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 04:16:00,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 04:16:01,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 04:16:01,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 04:16:02,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 04:16:02,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 04:16:02,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 04:16:03,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 04:16:03,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 04:16:03,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 04:16:04,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 04:16:04,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 04:16:05,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 04:16:05,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 04:16:05,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 04:16:06,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 04:16:06,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 04:16:07,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 04:16:07,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 04:16:07,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 04:16:08,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 04:16:08,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 04:16:08,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 04:16:09,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 04:16:09,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 04:16:10,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 04:16:10,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 04:16:10,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 04:16:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 04:16:11,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 04:16:11,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 04:16:12,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 04:16:12,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 04:16:13,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 04:16:13,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 04:16:13,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 04:16:14,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 04:16:14,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 04:16:14,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 04:16:15,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 04:16:15,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 04:16:16,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 04:16:16,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 04:16:16,849 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:16:16,849 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 140329.671875, u'ReplayWALOffset': u'12999:9392128', u'NTablesCreated': 151640, u'RecentReadIOAmpSet': 0.037234771996736526, u'ReadIOAmp': 154.68692016601562, u'BlockCacheMisses': 5420272, u'NFlushes': 75529, u'RecentWriteAmp': 2.1919779777526855, u'NumSyncFlushes': 2351, u'TotalMemUsed': 131464933, u'NBloomFilterHits': 58570787, u'Fragmentation': 0.2874490022659302, u'NumNormalFlushes': 48191, u'RecentBlockCacheHitRatio': 30.572858810424805, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 65065, u'ReadIOAmp': 1.0187622614588907, u'MemtablesMemUsed': 4908056, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 516277746, u'CheckpointOverhead': 696320, u'NItemsGet': 336420, u'NWriteIOs': 124811, u'NReadBytes': 2602634861L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 353687, u'SizeRatio': 0.007665959271517667, u'NWriteBytes': 190119936, u'BloomFilterMemUsed': 0, u'NumTables': 101, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 827392}, {u'TargetSize': 46137344, u'LogicalSize': 39122782, u'SizeRatio': 0.8479634631763805, u'NWriteBytes': 177463296, u'BloomFilterMemUsed': 0, u'NumTables': 22, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 1540096}, {u'TargetSize': 642545133, u'LogicalSize': 408230980, u'SizeRatio': 0.6353343275576565, u'NWriteBytes': 242532352, u'BloomFilterMemUsed': 0, u'NumTables': 138, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 26030080}, {u'TargetSize': 32767077514L, u'LogicalSize': 19197158447L, u'SizeRatio': 0.5858672760424807, u'NWriteBytes': 93433856, u'BloomFilterMemUsed': 0, u'NumTables': 912, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 190103552}, {u'TargetSize': 128509027510L, u'LogicalSize': 65534155031L, u'SizeRatio': 0.5099575983165885, u'NWriteBytes': 4485287936L, u'BloomFilterMemUsed': 0, u'NumTables': 463, u'WastedSpace': 1279641276, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 65534115840L}], u'NItemsGarbageCollected': 3028906, u'NInternalFragmentationCompacts': 1, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 401738089, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 168864, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 960973, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 496038, u'CompactWriter': {u'NDataBlockWrites': 349386, u'NWriteBytes': 839217152, u'NIndexBlockWrites': 7871, u'NBytesIncoming': 1446265506, u'NSyncs': 0, u'NInserts': 46208608, u'NWriteIOs': 5941}, u'NWriteBytes': 23343307250L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 342732, u'NReadBytes': 1095361442, u'NDataBlockReads': 336420, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1009260}, u'NBloomFilterMisses': 0, u'NItemsMerged': 625675, u'LogicalDiskSize': 65534155031L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.7254325196859283, u'BufferMemUsed': 502342, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 46336996584L, u'NTableFiles': 1708, u'NFileCountCompacts': 1195, u'TableObjectMemUsed': 436232, u'TreeSnapshotMemoryUsed': 204736, u'TotalDiskUsage': 85726515200L, u'BytesIncoming': 32178468178L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 1727, u'NItemsPut': 25624786, u'NSyncs': 27342, u'CompactReader': {u'NReadIOs': 612010, u'NReadBytes': 1503810470, u'NDataBlockReads': 581518, u'NBytesOutgoing': 0, u'NIndexBlockReads': 30492}, u'ActiveDiskUsage': 65752616960L, u'NTablesDeleted': 26355, u'IterateReader': {u'NReadIOs': 6231, u'NReadBytes': 3462949, u'NDataBlockReads': 3155, u'NBytesOutgoing': 0, u'NIndexBlockReads': 3076}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 6687384, u'NWriteBytes': 22504090098L, u'NIndexBlockWrites': 177319, u'NBytesIncoming': 28875509891L, u'NSyncs': 27342, u'NInserts': 21709543, u'NWriteIOs': 118870}}, u'NReadBytes': 686299431470L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 189.46665954589844, u'NInternalFragmentationCompacts': 1, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 428449136, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 12288, u'NTxns': 1722618, u'NWriteBytes': 28200828928L, u'NSegments': 1590, u'NSyncs': 1722618, u'NLoggedBytes': 28200828928L, u'NActiveSegments': 2, u'NWrites': 1817418, u'NInserts': 25657183, u'DiskUsed': 15925248}, u'NGets': 1560052, u'NReadBytesSet': 1246065421, u'NReadBytesCompact': 15873928229L, u'IndexResidentRatio': 0.041522663086652756, u'CompactQueueSize': 0, u'NSyncsPerSec': 3.299999952316284, u'BlockCacheHitRatio': 50.94109344482422, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 670422040292L, u'BloomFilterMemUsedRatio': 0.21568116545677185, u'NWriteBytes': 67664759283L, u'BytesOutgoing': 474256736, u'RecentReadIOAmp': 0.9147414565086365, u'WriteAmp': 1.9464116096496582, u'NBlocksCached': 26507, u'LogicalDataSize': 47960581244L, u'ActiveBloomFilterMemUsed': 43174752, u'NFileCountCompacts': 5658, u'NTableFiles': 2529, u'FlushQueueSize': 72, u'TableObjectMemUsed': 639840, u'BlockCacheQuota': 92525378, u'TotalDiskUsage': 87727751168L, u'NPurgerRuns': 45, u'ReadIOAmpSet': 0.0, u'NSyncs': 82026, u'NReadIOsSetPerSec': 18.133333206176758, u'NReadIOsPerSec': 354.70001220703125, u'NTablesDeleted': 112808, u'BloomFilterFPR': 0.018035300076007843, u'NBloomFilterHitsPerSec': 2308.033447265625, u'WriteCacheMemUsed': 7482440, u'RecentBloomFilterFPR': 0.020193016156554222, u'BytesPerRead': 428944.65625, u'BytesIncomingPerSec': 970870.125, u'NReadIOsGetPerSec': 55.43333435058594, u'NGetStats': 18472, u'TotalIndexBlocksSize': 543418145, u'NGetStatsPerSec': 0.9666666388511658, u'NWriteBytesCompact': 14985236480L, u'NReadIOsSet': 481247, u'ActiveDataSize': 47960581244L, u'NWriteBytesCompactPerSec': 768819.1875, u'NBloomFilterMissesPerSec': 47.56666564941406, u'NCommitBatches': 2029308, u'NGetStatsComputed': 18472, u'NWriteIOs': 2108803, u'NReadIOsGet': 241800875, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 23131344, u'LSMTreeObjectMemUsed': 182336, u'NSetsPerSec': 487.0, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 3.0, u'ManagedFileObjectMemUsed': 247680, u'ReadAmpGet': 1410.9993896484375, u'BlockCacheMemUsed': 88847328, u'NWriterCompacts': 0, u'NReadIOs': 247774142, u'WALBufferMemUsed': 65536, u'ReadAmp': 1444.4779052734375, u'keyStats': {u'NItemsTransformed': 308935548, u'NTablesCreated': 45628, u'ReadIOAmp': 28.071421321891965, u'MemtablesMemUsed': 369688, u'NBloomFilterHits': 58568544, u'TotalIndexBlocksSize': 27130191, u'CheckpointOverhead': 33501184, u'NItemsGet': 8601535, u'NWriteIOs': 98285, u'NReadBytes': 683620310319L, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 1061746, u'SizeRatio': 0.023012724789706143, u'NWriteBytes': 488230912, u'BloomFilterMemUsed': 10632, u'NumTables': 63, u'WastedSpace': 162, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 978944}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 33867661, u'SizeRatio': 0.7340617830103094, u'NWriteBytes': 6314618880L, u'BloomFilterMemUsed': 1140736, u'NumTables': 22, u'WastedSpace': 71669, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 33935360}, {u'TargetSize': 177399250, u'LogicalSize': 150539418, u'SizeRatio': 0.8485910622508269, u'NWriteBytes': 1032146944, u'BloomFilterMemUsed': 5006592, u'NumTables': 84, u'WastedSpace': 415074, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 150929408}, {u'TargetSize': 1773992579, u'LogicalSize': 1773992579, u'SizeRatio': 1.0, u'NWriteBytes': 679170048, u'BloomFilterMemUsed': 37015552, u'NumTables': 255, u'WastedSpace': 6198162, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1780187136}], u'NItemsGarbageCollected': 5433747, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 26708283, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 58848, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 246730447, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 895654, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 186969, u'CompactWriter': {u'NDataBlockWrites': 5099712, u'NWriteBytes': 14127382528L, u'NIndexBlockWrites': 66334, u'NBytesIncoming': 19392475836L, u'NSyncs': 0, u'NInserts': 360697618, u'NWriteIOs': 31809}, u'NWriteBytes': 15771322210L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 241457313, u'NReadBytes': 669325938221L, u'NDataBlockReads': 614172201, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1006726586}, u'NBloomFilterMisses': 1075594, u'NItemsMerged': 0, u'LogicalDiskSize': 1773992579, u'TotalBloomFilterMemUsed': 32163464, u'WriteAmp': 14.081628702598373, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 43173512, u'LogicalDataSize': 1623453161, u'NTableFiles': 613, u'NFileCountCompacts': 2188, u'TableObjectMemUsed': 152024, u'TreeSnapshotMemoryUsed': 92768, u'TotalDiskUsage': 1999532032, u'BytesIncoming': 1119992761, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 2921, u'NItemsPut': 22818585, u'NSyncs': 27342, u'CompactReader': {u'NReadIOs': 5273134, u'NReadBytes': 14294372098L, u'NDataBlockReads': 5169847, u'NBytesOutgoing': 0, u'NIndexBlockReads': 103287}, u'ActiveDiskUsage': 1966030848, u'NTablesDeleted': 45507, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.018033524099216592, u'Writer': {u'NDataBlockWrites': 338106, u'NWriteBytes': 1643939682, u'NIndexBlockWrites': 40106, u'NBytesIncoming': 1185801122, u'NSyncs': 27342, u'NInserts': 20412397, u'NWriteIOs': 66476}}, u'NReadBytesCompactPerSec': 768211.0, u'TableMetaMemUsed': 742735, u'NDeletes': 0, u'BlockCacheHitsPerSec': 83.43333435058594, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 1075743, u'LogicalDiskSize': 67308279109L, u'NSets': 6840509, u'TotalBloomFilterMemUsed': 32164704, u'BufferMemUsed': 502342, u'RecentReadAmp': 0.0, u'NGetsPerSec': 60.599998474121094, u'NInserts': 15978076, u'NCommitBatchesPerSec': 191.56666564941406, u'FragmentationWithOverhead': 0.4533020555973053, u'NCompactsPerSec': 0.6000000238418579, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 908540.6875, u'TreeSnapshotMemoryUsed': 340672, u'BytesIncoming': 34763847719L, u'NCompacts': 6923, u'NumPersistentFlushes': 24987, u'NWriteIOsPerSec': 150.6666717529297, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 2315.671142578125, u'ActiveDiskUsage': 67719098368L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.9666666388511658, u'NReadBytesSetPerSec': 46907.19921875, u'BlockCacheHits': 5628225, u'NWriteBytesPerSec': 2128125.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 40947, u'ReadIOAmp': 0.12761377613776137, u'MemtablesMemUsed': 2204696, u'NBloomFilterHits': 2243, u'TotalIndexBlocksSize': 10208, u'CheckpointOverhead': 1253376, u'NItemsGet': 6504, u'NWriteIOs': 68289, u'NReadBytes': 76486290, u'LevelStats': [{u'TargetSize': 46137344, u'LogicalSize': 446216, u'SizeRatio': 0.009671471335671165, u'NWriteBytes': 271941632, u'BloomFilterMemUsed': 536, u'NumTables': 44, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 360448}, {u'TargetSize': 46137344, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 46137344, u'LogicalSize': 131499, u'SizeRatio': 0.002850164066661488, u'NWriteBytes': 16015360, u'BloomFilterMemUsed': 704, u'NumTables': 11, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 90112}], u'NItemsGarbageCollected': 5789969, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 2764, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 19968, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 82722, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 59728, u'CompactWriter': {u'NDataBlockWrites': 2275, u'NWriteBytes': 18636800, u'NIndexBlockWrites': 2275, u'NBytesIncoming': 12456394, u'NSyncs': 0, u'NInserts': 120575, u'NWriteIOs': 2275}, u'NWriteBytes': 349300895, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 830, u'NReadBytes': 740629, u'NDataBlockReads': 1236, u'NBytesOutgoing': 0, u'NIndexBlockReads': 1236}, u'NBloomFilterMisses': 149, u'NItemsMerged': 0, u'LogicalDiskSize': 131499, u'TotalBloomFilterMemUsed': 1240, u'WriteAmp': 0.12062814680015128, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1240, u'LogicalDataSize': 131499, u'NTableFiles': 208, u'NFileCountCompacts': 2275, u'TableObjectMemUsed': 51584, u'TreeSnapshotMemoryUsed': 43168, u'TotalDiskUsage': 1703936, u'BytesIncoming': 2895683174L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 2275, u'NItemsPut': 5793822, u'NSyncs': 27342, u'CompactReader': {u'NReadIOs': 81892, u'NReadBytes': 75745661, u'NDataBlockReads': 40946, u'NBytesOutgoing': 0, u'NIndexBlockReads': 40946}, u'ActiveDiskUsage': 450560, u'NTablesDeleted': 40946, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.06229096989966555, u'Writer': {u'NDataBlockWrites': 38672, u'NWriteBytes': 330664095, u'NIndexBlockWrites': 38672, u'NBytesIncoming': 62826989, u'NSyncs': 27342, u'NInserts': 364644, u'NWriteIOs': 66014}}}}} 2021-12-16 04:16:16,854 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.67': [0.0027795331552624702, 0.006412389688193798, 0.006256026681512594, 0.004127239342778921, 0.003994441125541925, 0.0044576069340109825, 0.0042524333111941814, 0.004753243178129196, 0.004383909981697798, 0.004381217528134584, 0.004632345866411924, 0.004458579700440168, 0.004745154175907373, 0.004849925637245178, 0.004502692259848118, 0.006374937482178211, 0.005871925037354231, 0.0059201219119131565, 0.006204182747751474, 0.006210444029420614, 0.0060179284773766994, 0.006606325972825289, 0.004311813972890377, 0.004316831007599831, 0.004528461955487728, 0.004424918442964554, 0.004762335680425167, 0.004338521044701338, 0.004118669778108597, 0.004518086090683937, 0.004509063437581062, 0.004332923796027899, 0.00455492315813899, 0.004808661062270403, 0.004590973723679781, 0.004331489559262991, 0.004380186554044485, 0.00484473817050457, 0.004510371945798397, 0.004397754091769457, 0.004490402061492205, 0.002949393354356289, 0.0026826183311641216, 0.002773665590211749, 0.0026614530943334103, 0.002788584679365158, 0.0024693834129720926, 0.0024586161598563194, 0.0025554464664310217, 0.0026437018532305956, 0.0024876450188457966, 0.0029548003803938627, 0.002795265754684806, 0.0026074396446347237, 0.0027897031977772713, 0.0028993624728173018, 0.002734443172812462, 0.0025141173973679543, 0.0027365549467504025, 0.0027453028596937656, 0.002760145580396056, 0.002394316950812936, 0.00284265773370862, 0.0024366152938455343], '172.23.110.68': [0.004458312876522541, 0.004383636638522148, 0.00414810748770833, 0.004487319849431515, 0.004582071676850319, 0.004098675679415464, 0.0042733862064778805, 0.004424382001161575, 0.0047372072003781796, 0.004591827746480703, 0.004634673707187176, 0.004961308091878891, 0.004578498657792807, 0.004742160439491272, 0.004694477654993534, 0.004495400004088879, 0.0027129726950079203, 0.0028577966149896383, 0.0025116598699241877, 0.002608193550258875, 0.0027483117301017046, 0.002341720974072814, 0.0026952263433486223, 0.002691702451556921, 0.0028005042113363743, 0.002497568028047681, 0.002481709001585841, 0.002944387961179018, 0.002640168182551861, 0.0027901872526854277, 0.0024227106478065252, 0.0025942169595509768, 0.002413741545751691, 0.0026644801255315542, 0.00264723040163517, 0.002813525265082717, 0.0026597341056913137, 0.002715173875913024, 0.0027229723054915667, 0.004145585931837559, 0.006006390321999788, 0.003971028607338667, 0.0038275227416306734, 0.00387521437369287, 0.004148105159401894, 0.0042100525461137295, 0.004427946172654629, 0.004326705355197191, 0.00423648115247488, 0.004352292977273464, 0.0045693847350776196, 0.0042952257208526134, 0.004336134064942598, 0.004341709427535534, 0.004713213536888361, 0.004448326304554939, 0.004517066292464733, 0.004957353230565786, 0.004837494809180498, 0.004741475451737642, 0.004838718567043543, 0.002423073397949338, 0.002419056138023734, 0.0028682954143732786], '172.23.110.65': [0.2874490022659302, 0.26305899024009705, 0.21480850875377655, 0.2142000049352646, 0.28843066096305847, 0.2611173689365387, 0.2837681770324707, 0.2638263404369354, 0.3335707187652588, 0.34283629059791565, 0.3351726233959198, 0.3717135787010193, 0.37833884358406067, 0.3448794484138489, 0.3330460488796234, 0.33894965052604675, 0.3252970278263092, 0.33249735832214355, 0.3547922968864441, 0.33776789903640747, 0.31374239921569824, 0.330010324716568, 0.32855096459388733, 0.36930298805236816, 0.3572480082511902, 0.36286675930023193, 0.3155989348888397, 0.284580260515213, 0.26543697714805603, 0.35542839765548706, 0.34260910749435425, 0.3582323491573334, 0.284783273935318, 0.29817378520965576, 0.29716789722442627, 0.3481389582157135, 0.3008866012096405, 0.3102690577507019, 0.295831561088562, 0.31248822808265686, 0.34817731380462646, 0.31183379888534546, 0.3439352810382843, 0.3152863383293152, 0.3055911362171173, 0.29186463356018066, 0.2510690689086914, 0.29935166239738464, 0.30220937728881836, 0.30296140909194946, 0.26928290724754333, 0.3633827269077301, 0.28932228684425354, 0.3056625723838806, 0.31643128395080566, 0.3074268102645874, 0.3679144084453583, 0.3090295195579529, 0.3071290850639343, 0.36401328444480896, 0.32978716492652893, 0.3082217872142792, 0.32558947801589966, 0.24803458154201508], '172.23.110.64': [0.3677065968513489, 0.3977845013141632, 0.4029616713523865, 0.38644129037857056, 0.3880358934402466, 0.32948243618011475, 0.36512789130210876, 0.3186211884021759, 0.35062310099601746, 0.3441051244735718, 0.35248422622680664, 0.361237496137619, 0.361899733543396, 0.3266145586967468, 0.4005531370639801, 0.3865322172641754, 0.34132957458496094, 0.35058125853538513, 0.30298516154289246, 0.4050237536430359, 0.3481055498123169, 0.36472082138061523, 0.37735551595687866, 0.3436582684516907, 0.3612990379333496, 0.3479396104812622, 0.28516557812690735, 0.34621956944465637, 0.3518694341182709, 0.39182770252227783, 0.39547014236450195, 0.34666088223457336, 0.3650973439216614, 0.39860525727272034, 0.3420349657535553, 0.3623550236225128, 0.3679356276988983, 0.3376622796058655, 0.40609121322631836, 0.36252710223197937, 0.40137234330177307, 0.34818288683891296, 0.32457780838012695, 0.3560466766357422, 0.36448878049850464, 0.28337404131889343, 0.3110818862915039, 0.3962773382663727, 0.356316477060318, 0.3748188316822052, 0.4051640033721924, 0.3990670144557953, 0.3856521546840668, 0.3901585042476654, 0.39141908288002014, 0.24816085398197174, 0.41449832916259766, 0.33229175209999084, 0.37407079339027405, 0.39751705527305603, 0.2916548252105713, 0.3648710548877716, 0.39171910285949707, 0.35590529441833496]} 2021-12-16 04:16:17,069 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 20.0 2021-12-16 04:16:17,296 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.68 is 20.0 2021-12-16 04:16:17,543 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.67 is 20.0 2021-12-16 04:16:17,862 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 20.0 2021-12-16 04:16:17,862 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.67': 20.0, '172.23.110.68': 20.0, '172.23.110.65': 20.0, '172.23.110.64': 20.0} ############################################################ # # Step 9: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 4680.51953125 # ############################################################ 2021-12-16 04:16:17,917 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:16:17,917 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 76 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-16 04:17:33,927 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:17:33,928 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:17:38,930 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:17:39,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:39,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:17:39,326 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:39,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:17:39,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:17:39,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:17:39,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:17:39,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:17:39,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:17:39,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:17:39,404 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:17:39,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:17:39,453 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['162533 memcached\n'], []) 2021-12-16 04:17:39,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:17:39,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:17:39,542 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['162533 memcached\n'], []) 2021-12-16 04:17:39,542 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:17:39,546 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:17:39,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:39,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:17:40,016 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:40,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:17:40,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:17:40,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:17:40,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:17:40,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:17:40,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:17:40,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:17:40,134 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:17:40,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:17:40,193 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['132136 memcached\n'], []) 2021-12-16 04:17:40,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:17:40,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:17:40,286 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['132136 memcached\n'], []) 2021-12-16 04:17:40,288 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:17:40,292 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:17:40,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:40,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:17:40,769 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:40,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:17:40,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:17:40,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:17:40,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:17:40,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:17:40,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:17:40,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:17:40,877 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:17:40,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:17:40,927 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['231496 memcached\n'], []) 2021-12-16 04:17:40,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:17:41,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:17:41,071 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['231496 memcached\n'], []) 2021-12-16 04:17:41,071 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:17:41,072 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:17:41,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:41,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:17:41,529 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:41,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:17:41,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:17:41,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:17:41,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:17:41,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:17:41,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:17:41,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:17:41,713 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:17:41,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:17:41,802 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['222131 memcached\n'], []) 2021-12-16 04:17:41,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:17:41,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:17:41,957 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['222131 memcached\n'], []) 2021-12-16 04:17:41,957 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:17:41,960 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:17:42,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:43,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:17:43,410 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:43,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:17:43,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:17:43,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:17:43,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:17:43,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:17:43,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:17:43,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:17:43,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:17:43,556 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:17:43,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:17:43,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:43,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:43,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:43,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:43,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:43,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:43,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:43,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:43,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:17:43,838 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:17:43,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:17:43,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:43,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:17:43,875 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:17:43,875 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:17:43,878 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:17:43,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:44,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:17:44,336 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:44,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:17:44,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:17:44,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:17:44,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:17:44,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:17:44,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:17:44,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:17:44,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:17:44,509 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:17:44,535 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:17:44,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:44,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:44,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:44,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:44,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:44,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:44,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:44,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:44,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:44,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:44,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:44,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:44,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:44,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:44,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:45,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:45,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:45,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:45,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:45,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:45,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:17:45,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:17:45,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:17:45,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:17:45,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:17:45,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:17:45,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:17:45,617 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:17:45,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:17:45,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:17:45,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:17:45,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:17:45,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:17:45,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:17:45,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:45,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:45,987 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:17:46,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:17:46,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:46,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:46,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:17:46,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:17:46,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:46,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:46,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:17:46,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:17:46,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:46,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:46,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:17:46,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:17:46,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:46,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:46,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:17:46,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:17:46,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:17:46,520 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:17:46,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:17:46,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:46,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:17:46,576 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:17:46,576 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:17:46,581 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:17:46,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:47,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:17:47,095 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:47,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:17:47,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:17:47,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:17:47,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:17:47,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:17:47,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:17:47,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:17:47,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:17:47,240 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:17:47,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:17:47,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:47,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:47,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:47,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:47,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:47,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:47,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:47,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:47,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:47,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:47,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:47,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:47,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:47,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:47,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,005 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:17:48,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:17:48,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:17:48,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:17:48,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:17:48,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:17:48,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:17:48,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:17:48,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,397 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:17:48,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:17:48,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:17:48,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:17:48,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:17:48,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:17:48,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:17:48,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:17:48,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:17:48,747 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:17:48,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:17:48,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:17:48,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:48,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:17:48,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:17:48,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:17:48,951 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:17:48,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:17:48,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:48,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:17:48,990 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:17:48,990 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:17:48,993 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:17:49,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:49,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:17:49,486 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:49,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:17:49,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:17:49,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:17:49,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:17:49,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:17:49,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:17:49,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:17:49,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:17:49,644 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:17:49,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:17:49,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:49,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:49,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:49,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:17:49,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:49,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:49,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:49,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:17:49,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:49,984 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:50,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:50,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:17:50,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:50,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:50,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:50,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:17:50,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:50,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:50,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:50,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:17:50,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:50,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:17:50,255 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:50,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:17:50,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:17:50,302 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:17:50,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:17:50,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:17:50,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:17:50,338 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:17:50,338 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:17:50,339 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:17:50,342 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:17:50,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:17:51,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:17:51,523 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:17:51,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:17:51,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:17:51,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:17:51,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:17:51,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:17:51,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:17:51,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:17:51,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:17:53,239 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-16 04:17:53,243 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:17:53,244 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:17:55,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:17:55,407 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:17:55,415 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:17:55,415 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:17:57,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:17:57,582 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:17:57,589 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:17:57,589 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:17:59,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:17:59,770 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:17:59,776 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:17:59,776 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:01,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:01,941 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:01,948 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:01,948 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:03,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:04,089 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:04,095 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:04,095 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:06,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:06,273 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:06,276 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:06,276 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:08,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:08,453 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:08,457 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:08,457 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:10,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:10,634 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:10,638 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:10,638 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:12,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:12,801 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:12,805 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:12,805 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:14,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:14,974 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:14,979 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:14,979 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:16,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:17,148 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:18:17,151 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:17,151 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:19,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:19,288 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:19,289 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:21,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:21,434 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:21,434 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:23,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:23,582 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:23,584 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:25,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:25,723 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:25,723 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:27,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:27,884 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:27,884 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:29,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:30,032 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:30,032 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:32,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:32,196 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:32,196 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:34,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:34,338 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:34,339 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:36,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:36,492 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:36,492 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:38,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:38,674 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:38,674 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:40,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:40,831 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:40,831 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:42,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:42,973 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:42,974 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:44,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:45,135 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:45,135 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:47,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:47,292 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:47,292 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:49,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:49,448 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:49,448 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:51,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:51,609 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:51,609 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:53,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:53,773 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:53,775 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:55,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:55,928 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:55,928 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:18:57,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:18:58,085 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:18:58,085 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:00,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:00,242 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:00,243 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:02,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:02,398 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:02,398 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:04,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:04,559 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:04,559 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:06,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:06,719 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:06,719 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:08,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:08,877 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:08,877 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:10,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:11,045 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:11,045 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:13,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:13,200 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:13,200 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:15,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:15,434 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:15,434 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:17,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:17,704 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:17,704 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:19,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:19,936 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:19,936 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:21,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:22,117 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:22,118 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:24,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:24,293 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:24,293 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:26,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:26,678 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:26,680 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:28,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:28,894 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:28,894 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:30,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:31,138 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:31,140 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:19:33,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:33,526 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:19:33,526 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 103.186000109. 2021-12-16 04:19:33,526 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:19:33,529 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:19:33,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:19:33,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:19:34,022 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:19:34,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:19:34,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:19:34,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:19:34,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:19:34,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:19:34,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:19:34,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:19:34,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:34,388 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:19:34,388 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.861999988556. 2021-12-16 04:19:34,390 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:19:34,391 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:19:34,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:19:34,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:19:34,884 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:19:34,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:19:34,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:19:34,926 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:19:34,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:19:34,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:19:35,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:19:35,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:19:35,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:35,244 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:19:35,244 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855999946594. 2021-12-16 04:19:35,246 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:19:35,247 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:19:35,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:19:35,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:19:35,585 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:19:35,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:19:35,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:19:35,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:19:35,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:19:35,650 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:19:35,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:19:35,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:19:35,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:35,857 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:35,857 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:37,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:38,020 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:38,020 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:40,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:40,183 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:40,183 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:42,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:42,822 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:42,823 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:44,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:45,085 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:45,085 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:47,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:48,051 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:48,051 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:50,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:50,365 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:19:50,365 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:19:52,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:19:53,048 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:19:53,048 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 17.8020000458. ############################################################ # # RAM FootPrint: 4646.18359375 # ############################################################ 2021-12-16 04:19:53,081 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:19:53,081 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 115 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-16 04:21:48,085 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:21:48,085 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:21:53,086 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:21:53,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:21:53,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:21:53,684 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:21:53,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:21:53,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:21:53,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:21:53,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:21:53,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:21:53,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:21:53,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:21:53,778 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:21:53,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:21:53,834 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['190547 memcached\n'], []) 2021-12-16 04:21:53,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:21:53,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:21:53,921 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['190547 memcached\n'], []) 2021-12-16 04:21:53,921 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:21:53,924 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:21:54,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:21:54,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:21:54,401 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:21:54,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:21:54,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:21:54,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:21:54,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:21:54,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:21:54,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:21:54,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:21:54,546 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:21:54,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:21:54,618 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['161485 memcached\n'], []) 2021-12-16 04:21:54,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:21:54,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:21:54,721 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['161485 memcached\n'], []) 2021-12-16 04:21:54,721 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:21:54,726 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:21:54,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:21:55,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:21:55,167 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:21:55,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:21:55,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:21:55,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:21:55,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:21:55,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:21:55,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:21:55,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:21:55,283 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:21:55,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:21:55,365 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['36161 memcached\n'], []) 2021-12-16 04:21:55,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:21:55,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:21:55,477 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['36161 memcached\n'], []) 2021-12-16 04:21:55,479 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:21:55,482 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:21:55,859 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:21:56,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:21:56,214 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:21:56,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:21:56,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:21:56,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:21:56,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:21:56,318 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:21:56,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:21:56,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:21:56,378 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:21:56,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:21:56,463 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['234599 memcached\n'], []) 2021-12-16 04:21:56,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:21:56,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:21:56,602 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['234599 memcached\n'], []) 2021-12-16 04:21:56,602 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:21:56,605 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:21:56,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:22:00,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:22:00,137 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:22:00,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:22:00,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:22:00,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:22:00,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:22:00,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:22:00,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:22:00,255 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:22:00,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:22:00,295 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:22:00,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:22:00,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:00,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:00,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:00,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:00,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:00,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:00,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:00,502 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:00,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:22:00,529 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:22:00,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:22:00,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:00,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:22:00,569 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:22:00,569 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:22:00,573 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:22:00,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:22:00,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:22:01,023 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:22:01,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:22:01,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:22:01,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:22:01,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:22:01,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:22:01,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:22:01,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:22:01,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:22:01,173 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:22:01,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:22:01,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:01,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:01,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:01,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:01,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:01,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:01,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:01,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:01,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:01,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:01,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:01,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:01,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:02,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:02,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:02,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:02,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:22:02,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:22:02,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:22:02,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:22:02,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,487 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:22:02,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:22:02,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:22:02,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:22:02,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:22:02,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:22:02,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:22:02,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:22:02,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:02,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:02,984 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:22:03,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:22:03,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:03,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:03,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:22:03,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:22:03,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:03,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:03,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:22:03,278 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:22:03,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:03,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:03,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:22:03,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:22:03,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:03,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:03,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:22:03,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:22:03,566 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:22:03,589 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:22:03,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:22:03,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:03,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:22:03,648 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:22:03,648 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:22:03,651 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:22:03,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:22:04,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:22:04,148 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:22:04,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:22:04,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:22:04,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:22:04,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:22:04,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:22:04,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:22:04,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:22:04,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:22:04,292 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:22:04,318 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:22:04,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:04,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:04,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:04,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:04,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:04,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:04,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:04,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:04,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:04,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:04,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:04,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:04,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:04,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:04,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:04,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:04,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:04,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:04,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:04,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:05,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:05,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:05,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:05,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:05,255 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:22:05,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:22:05,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:22:05,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:22:05,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:22:05,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:22:05,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:22:05,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:22:05,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:22:05,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:22:05,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:05,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:22:05,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:22:05,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:05,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:06,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:22:06,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:22:06,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:06,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:06,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:22:06,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:22:06,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:06,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:06,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:22:06,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:22:06,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:06,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:06,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:22:06,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:22:06,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:22:06,507 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:22:06,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:22:06,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:06,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:22:06,569 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:22:06,569 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:22:06,572 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:22:06,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:22:06,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:22:07,084 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:22:07,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:22:07,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:22:07,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:22:07,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:22:07,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:22:07,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:22:07,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:22:07,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:22:07,250 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:22:07,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:22:07,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:07,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:07,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:07,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:22:07,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:07,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:07,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:07,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:22:07,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:07,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:07,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:07,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:22:07,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:07,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:07,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:07,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:22:07,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:07,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:07,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:07,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:22:07,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:08,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:08,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:08,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:22:08,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:08,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:22:08,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:08,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:22:08,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:22:08,244 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:22:08,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:22:08,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:22:08,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:22:08,309 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:22:08,309 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:22:08,311 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:22:08,313 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:22:08,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:22:08,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:22:08,834 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:22:08,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:22:08,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:22:08,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:22:08,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:22:08,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:22:08,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:22:08,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:22:08,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:09,168 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:09,174 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:09,174 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:11,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:11,342 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:11,346 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:11,346 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:13,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:13,533 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:13,538 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:13,538 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:15,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:15,694 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:15,697 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:15,698 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:17,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:17,885 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:17,891 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:17,891 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:19,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:20,048 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:20,052 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:20,052 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:22,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:22,223 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:22,226 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:22,226 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:24,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:24,411 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:24,414 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:24,415 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:26,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:26,591 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:26,595 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:26,595 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:28,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:28,766 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:22:28,770 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:28,770 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:30,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:30,911 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:30,911 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:32,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:33,065 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:33,065 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:35,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:35,223 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:35,223 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:37,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:37,391 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:37,391 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:39,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:39,575 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:39,575 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:41,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:41,752 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:41,753 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:43,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:43,901 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:43,901 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:45,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:46,046 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:46,046 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:48,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:48,198 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:48,198 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:50,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:50,367 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:50,368 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:52,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:52,519 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:52,519 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:54,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:54,677 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:54,677 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:56,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:56,825 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:56,825 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:22:58,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:22:58,986 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:22:58,987 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:00,987 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:01,135 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:01,135 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:03,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:03,295 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:03,295 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:05,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:05,454 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:05,456 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:07,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:07,612 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:07,612 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:09,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:09,773 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:09,773 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:11,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:11,933 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:11,934 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:13,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:14,086 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:14,086 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:16,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:16,244 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:16,246 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:18,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:18,417 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:18,417 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:20,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:20,579 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:20,581 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:22,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:22,798 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:22,799 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:24,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:25,528 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:25,528 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:27,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:28,141 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:28,141 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:30,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:30,407 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:30,407 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:32,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:33,392 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:33,394 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:35,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:35,894 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:35,894 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:37,895 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:38,296 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:38,296 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:40,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:40,816 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:40,816 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:42,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:43,480 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:43,480 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:23:45,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:46,081 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:23:46,082 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 97.7709999084. 2021-12-16 04:23:46,082 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:23:46,085 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:23:46,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:23:46,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:23:46,595 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:23:46,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:23:46,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:23:46,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:23:46,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:23:46,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:23:46,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:23:46,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:23:46,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:46,944 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:23:46,944 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.861000061035. 2021-12-16 04:23:46,946 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:23:46,947 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:23:47,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:23:47,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:23:47,466 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:23:47,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:23:47,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:23:47,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:23:47,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:23:47,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:23:47,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:23:47,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:23:47,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:47,822 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:23:47,822 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.878000020981. 2021-12-16 04:23:47,823 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:23:47,825 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:23:48,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:23:49,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:23:50,049 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:23:50,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:23:50,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:23:50,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:23:50,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:23:50,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:23:50,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:23:50,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:23:50,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:50,326 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:50,326 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:23:52,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:53,167 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:53,167 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:23:55,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:56,122 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:56,124 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:23:58,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:23:59,802 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:23:59,802 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 04:24:01,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:24:04,237 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:24:04,237 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 16.4140000343. ############################################################ # # RAM FootPrint: 4544.859375 # ############################################################ 2021-12-16 04:24:04,269 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:24:04,269 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-16 04:25:49,273 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:25:49,273 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:25:54,275 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:25:54,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:25:54,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:25:54,822 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:25:54,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:25:54,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:25:54,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:25:54,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:25:54,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:25:54,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:25:54,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:25:54,957 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:25:54,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:25:55,020 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['191845 memcached\n'], []) 2021-12-16 04:25:55,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:25:55,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:25:55,132 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['191845 memcached\n'], []) 2021-12-16 04:25:55,132 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:25:55,135 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:25:55,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:25:55,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:25:55,612 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:25:55,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:25:55,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:25:55,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:25:55,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:25:55,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:25:55,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:25:55,744 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:25:55,766 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:25:55,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:25:55,828 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['162177 memcached\n'], []) 2021-12-16 04:25:55,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:25:55,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:25:55,971 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['162177 memcached\n'], []) 2021-12-16 04:25:55,973 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:25:55,976 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:25:56,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:25:56,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:25:56,490 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:25:56,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:25:56,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:25:56,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:25:56,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:25:56,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:25:56,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:25:56,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:25:56,644 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:25:56,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:25:56,723 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['37015 memcached\n'], []) 2021-12-16 04:25:56,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:25:56,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:25:56,868 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['37015 memcached\n'], []) 2021-12-16 04:25:56,868 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:25:56,871 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:25:57,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:25:57,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:25:57,371 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:25:57,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:25:57,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:25:57,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:25:57,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:25:57,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:25:57,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:25:57,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:25:57,608 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:25:57,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:25:57,686 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['236273 memcached\n'], []) 2021-12-16 04:25:57,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:25:57,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:25:57,838 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['236273 memcached\n'], []) 2021-12-16 04:25:57,838 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:25:57,841 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:25:57,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:26:03,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:26:03,990 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:26:03,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:26:04,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:26:04,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:26:04,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:26:04,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:26:04,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:26:04,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:26:04,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:26:04,112 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:26:04,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:26:04,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:04,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:04,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:04,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:04,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:04,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:04,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:04,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:04,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:26:04,451 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:26:04,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:26:04,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:04,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:26:04,506 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:26:04,506 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:26:04,509 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:26:04,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:26:04,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:26:04,956 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:26:04,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:26:04,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:26:04,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:26:05,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:26:05,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:26:05,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:26:05,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:26:05,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:26:05,125 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:26:05,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:26:05,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:05,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:05,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:05,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:05,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:05,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:05,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:05,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:05,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:05,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:05,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:05,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:05,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:05,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:06,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:06,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:06,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:26:06,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:26:06,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:26:06,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:26:06,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:26:06,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:26:06,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:26:06,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:26:06,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:26:06,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:26:06,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:06,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:06,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:26:06,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:26:06,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:07,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:26:07,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:26:07,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,145 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:07,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:26:07,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:26:07,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,249 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:07,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:26:07,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:26:07,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:07,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:26:07,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:26:07,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:07,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:26:07,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:26:07,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:26:07,563 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:26:07,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:26:07,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:07,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:26:07,615 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:26:07,615 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:26:07,618 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:26:07,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:26:08,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:26:08,124 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:26:08,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:26:08,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:26:08,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:26:08,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:26:08,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:26:08,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:26:08,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:26:08,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:26:08,286 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:26:08,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:26:08,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:08,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:08,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:08,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:08,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:08,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:08,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:08,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:08,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:08,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:08,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:08,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:08,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:08,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:08,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:08,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:08,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:08,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:08,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:09,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:09,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:09,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:09,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:09,249 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:09,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:26:09,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:26:09,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:26:09,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:26:09,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,553 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:26:09,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:26:09,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:26:09,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:26:09,739 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:26:09,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:26:09,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:09,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:09,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:26:09,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:26:09,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:10,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:26:10,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:26:10,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:10,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:26:10,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:26:10,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:10,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:26:10,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:26:10,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:10,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:26:10,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:26:10,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:10,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:26:10,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:26:10,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:26:10,674 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:26:10,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:26:10,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:10,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:26:10,736 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:26:10,736 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:26:10,743 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:26:10,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:26:11,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:26:11,562 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:26:11,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:26:11,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:26:11,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:26:11,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:26:11,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:26:11,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:26:11,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:26:11,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:26:12,015 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:26:12,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:26:12,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:12,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:12,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:12,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:26:12,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:12,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:12,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:12,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:26:12,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:12,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:12,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:12,808 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:26:12,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:12,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:12,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:12,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:26:13,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:13,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:13,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:13,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:26:13,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:13,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:13,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:13,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:26:13,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:13,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:26:13,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:13,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:26:13,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:26:13,505 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:26:13,505 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:26:13,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:26:13,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:26:13,640 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:26:13,640 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:26:13,640 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:26:13,644 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:26:13,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:26:14,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:26:14,155 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:26:14,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:26:14,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:26:14,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:26:14,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:26:14,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:26:14,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:26:14,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:26:14,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:14,450 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:26:14,457 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:14,457 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:16,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:16,622 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:26:16,630 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:16,630 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:18,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:18,798 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:26:18,802 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:18,802 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:20,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:20,967 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:26:20,971 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:20,971 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:22,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:23,154 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:26:23,158 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:23,158 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:25,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:25,321 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:25,322 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:27,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:27,474 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:27,476 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:29,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:29,625 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:29,625 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:31,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:31,773 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:31,773 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:33,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:33,918 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:33,918 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:35,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:36,068 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:36,068 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:38,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:38,213 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:38,214 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:40,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:40,410 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:40,410 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:42,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:42,569 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:42,569 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:44,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:44,733 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:44,733 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:46,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:46,891 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:46,891 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:48,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:49,055 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:49,055 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:51,056 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:51,207 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:51,207 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:53,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:53,358 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:53,359 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:55,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:55,529 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:55,529 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:57,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:57,680 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:57,680 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:26:59,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:26:59,842 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:26:59,842 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:01,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:01,993 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:01,993 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:03,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:04,151 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:04,151 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:06,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:06,309 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:06,309 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:08,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:08,467 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:08,469 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:10,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:10,628 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:10,628 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:12,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:12,786 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:12,788 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:14,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:15,036 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:15,036 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:17,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:17,200 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:17,200 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:19,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:20,026 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:20,026 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:22,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:22,194 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:22,194 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:24,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:24,390 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:24,390 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:26,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:26,578 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:26,578 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:28,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:28,750 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:28,750 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:30,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:30,915 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:30,915 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:32,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:33,088 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:33,088 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:35,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:35,549 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:27:35,549 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:27:37,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:38,019 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:27:38,019 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 84.3789999485. 2021-12-16 04:27:38,020 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:27:38,023 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:27:38,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:27:38,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:27:38,507 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:27:38,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:27:38,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:27:38,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:27:38,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:27:38,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:27:38,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:27:38,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:27:38,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:38,770 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:27:38,770 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.75. 2021-12-16 04:27:38,772 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:27:38,773 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:27:38,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:27:39,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:27:39,295 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:27:39,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:27:39,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:27:39,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:27:39,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:27:39,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:27:39,413 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:27:39,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:27:39,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:39,681 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:27:39,681 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.909999847412. 2021-12-16 04:27:39,683 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:27:39,684 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:27:40,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:27:40,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:27:41,065 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:27:41,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:27:41,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:27:41,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:27:41,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:27:41,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:27:41,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:27:41,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:27:41,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:27:41,845 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:27:41,845 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.16299986839. ############################################################ # # RAM FootPrint: 4520.234375 # ############################################################ 2021-12-16 04:27:41,875 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:27:41,877 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 117 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-16 04:29:38,881 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:29:38,881 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:29:43,884 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:29:44,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:44,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:29:44,447 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:44,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:29:44,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:29:44,487 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:29:44,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:29:44,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:29:44,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:29:44,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:29:44,598 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:29:44,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:29:44,707 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['193491 memcached\n'], []) 2021-12-16 04:29:44,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:29:44,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:29:44,798 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['193491 memcached\n'], []) 2021-12-16 04:29:44,798 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:29:44,802 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:29:44,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:45,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:29:45,263 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:45,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:29:45,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:29:45,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:29:45,318 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:29:45,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:29:45,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:29:45,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:29:45,388 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:29:45,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:29:45,443 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['163146 memcached\n'], []) 2021-12-16 04:29:45,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:29:45,487 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:29:45,555 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['163146 memcached\n'], []) 2021-12-16 04:29:45,556 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:29:45,561 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:29:45,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:45,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:29:45,987 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:45,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:29:46,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:29:46,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:29:46,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:29:46,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:29:46,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:29:46,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:29:46,098 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:29:46,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:29:46,163 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['37854 memcached\n'], []) 2021-12-16 04:29:46,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:29:46,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:29:46,257 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['37854 memcached\n'], []) 2021-12-16 04:29:46,257 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:29:46,260 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:29:46,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:46,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:29:46,792 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:46,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:29:46,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:29:46,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:29:46,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:29:46,895 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:29:46,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:29:46,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:29:46,966 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:29:46,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:29:47,109 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['237832 memcached\n'], []) 2021-12-16 04:29:47,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:29:47,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:29:47,250 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['237832 memcached\n'], []) 2021-12-16 04:29:47,250 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:29:47,253 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:29:47,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:48,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:29:48,283 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:48,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:29:48,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:29:48,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:29:48,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:29:48,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:29:48,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:29:48,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:29:48,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:29:48,446 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:29:48,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:29:48,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:48,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:48,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:48,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:48,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:48,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:48,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:48,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:48,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:29:48,785 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:29:48,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:29:48,801 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:48,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:29:48,838 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:29:48,839 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:29:48,842 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:29:48,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:49,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:29:49,311 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:49,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:29:49,329 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:29:49,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:29:49,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:29:49,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:29:49,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:29:49,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:29:49,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:29:49,444 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:29:49,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:29:49,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:49,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:49,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:49,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:49,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:49,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:49,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:49,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:49,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:49,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:49,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:49,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:49,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:49,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:49,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:49,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:50,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:50,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:50,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:50,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:50,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:50,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:50,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:29:50,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:29:50,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:29:50,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:29:50,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:29:50,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:29:50,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:29:50,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:29:50,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:29:50,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:29:50,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:29:50,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:29:50,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:50,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:50,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:29:50,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:29:51,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:51,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:51,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:29:51,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:29:51,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:51,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:51,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:29:51,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:29:51,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:51,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:51,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:29:51,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:29:51,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:51,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:51,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:29:51,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:29:51,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:29:51,529 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:29:51,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:29:51,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:51,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:29:51,598 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:29:51,598 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:29:51,601 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:29:51,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:52,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:29:52,105 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:52,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:29:52,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:29:52,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:29:52,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:29:52,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:29:52,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:29:52,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:29:52,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:29:52,246 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:29:52,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:29:52,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:52,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:52,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:52,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:52,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:52,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:52,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:52,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:52,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:52,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:52,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:52,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:52,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:53,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:53,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:53,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:53,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:29:53,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:29:53,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:29:53,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:29:53,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:29:53,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:29:53,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:29:53,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:29:53,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:29:53,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:29:53,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:29:53,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:29:53,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:29:53,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:29:53,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,801 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:29:53,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:29:53,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:29:53,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:29:53,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:53,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:53,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:29:53,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:29:54,005 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:54,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:54,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:29:54,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:29:54,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:29:54,065 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:29:54,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:29:54,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:54,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:29:54,098 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:29:54,098 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:29:54,101 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:29:54,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:54,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:29:54,559 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:54,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:29:54,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:29:54,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:29:54,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:29:54,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:29:54,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:29:54,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:29:54,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:29:54,709 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:29:54,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:29:54,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:54,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:54,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:54,867 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:29:54,895 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:54,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:54,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:54,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:29:55,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:55,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:55,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:29:55,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:55,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:55,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:29:55,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:55,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:55,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:29:55,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:55,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:55,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:29:55,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:29:55,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:55,632 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:29:55,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:29:55,673 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:29:55,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:29:55,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:29:55,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:29:55,730 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:29:55,730 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:29:55,730 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:29:55,746 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:29:55,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:29:56,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:29:56,270 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:29:56,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:29:56,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:29:56,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:29:56,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:29:56,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:29:56,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:29:56,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:29:56,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:29:56,549 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:29:56,555 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:29:56,556 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:29:58,556 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:29:58,707 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:29:58,711 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:29:58,713 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:00,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:00,881 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:00,884 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:00,885 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:02,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:03,069 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:03,075 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:03,075 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:05,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:05,239 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:05,243 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:05,243 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:07,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:07,420 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:07,424 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:07,424 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:09,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:09,607 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:09,612 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:09,612 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:11,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:11,778 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:11,782 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:11,782 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:13,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:13,944 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:13,947 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:13,947 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:15,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:16,112 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:30:16,117 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:16,117 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:18,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:18,263 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:18,265 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:20,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:20,415 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:20,415 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:22,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:22,562 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:22,562 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:24,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:24,726 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:24,726 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:26,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:26,885 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:26,885 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:28,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:29,029 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:29,029 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:31,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:31,188 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:31,188 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:33,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:33,329 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:33,331 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:35,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:35,532 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:35,533 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:37,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:37,694 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:37,694 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:39,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:39,855 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:39,855 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:41,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:42,009 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:42,009 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:44,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:44,168 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:44,168 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:46,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:46,328 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:46,328 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:48,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:48,482 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:48,482 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:50,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:50,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:50,641 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:52,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:52,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:52,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:54,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:54,947 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:54,947 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:56,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:57,107 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:57,107 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:30:59,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:30:59,263 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:30:59,263 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:01,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:01,418 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:01,418 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:03,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:03,572 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:03,573 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:05,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:05,733 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:05,733 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:07,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:07,910 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:07,910 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:09,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:10,088 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:10,088 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:12,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:12,256 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:12,256 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:14,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:14,443 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:14,443 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:16,443 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:16,611 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:16,611 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:18,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:18,789 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:18,789 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:20,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:20,953 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:20,953 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:22,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:23,122 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:23,124 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:25,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:25,308 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:25,309 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:27,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:27,457 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:31:27,457 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:31:29,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:29,619 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:31:29,619 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 93.888999939. 2021-12-16 04:31:29,621 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:31:29,622 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:31:29,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:31:30,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:31:30,105 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:31:30,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:31:30,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:31:30,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:31:30,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:31:30,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:31:30,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:31:30,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:31:30,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:30,457 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:31:30,459 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.838000059128. 2021-12-16 04:31:30,459 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:31:30,461 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:31:30,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:31:30,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:31:30,973 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:31:30,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:31:30,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:31:31,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:31:31,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:31:31,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:31:31,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:31:31,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:31:31,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:31,319 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:31:31,319 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.859999895096. 2021-12-16 04:31:31,319 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:31:31,322 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:31:31,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:31:31,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:31:31,773 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:31:31,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:31:31,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:31:31,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:31:31,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:31:31,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:31:31,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:31:31,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:31:31,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:31:32,151 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:31:32,151 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832000017166. ############################################################ # # RAM FootPrint: 4453.84375 # ############################################################ 2021-12-16 04:31:32,184 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:31:32,184 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 115 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-16 04:33:27,187 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:33:27,187 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:33:32,190 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:33:32,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:32,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:33:32,967 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:32,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:33:32,987 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:33:33,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:33:33,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:33:33,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:33:33,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:33:33,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:33:33,157 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:33:33,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:33:33,224 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['195851 memcached\n'], []) 2021-12-16 04:33:33,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:33:33,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:33:33,398 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['195851 memcached\n'], []) 2021-12-16 04:33:33,398 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:33:33,401 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:33:33,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:33,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:33:33,895 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:33,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:33:33,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:33:33,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:33:33,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:33:33,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:33:34,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:33:34,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:33:34,049 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:33:34,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:33:34,108 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['164120 memcached\n'], []) 2021-12-16 04:33:34,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:33:34,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:33:34,252 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['164120 memcached\n'], []) 2021-12-16 04:33:34,252 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:33:34,255 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:33:34,397 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:34,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:33:34,795 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:34,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:33:34,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:33:34,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:33:34,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:33:34,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:33:34,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:33:34,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:33:34,943 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:33:34,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:33:35,017 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['38670 memcached\n'], []) 2021-12-16 04:33:35,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:33:35,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:33:35,115 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['38670 memcached\n'], []) 2021-12-16 04:33:35,115 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:33:35,118 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:33:35,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:35,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:33:35,569 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:35,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:33:35,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:33:35,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:33:35,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:33:35,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:33:35,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:33:35,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:33:35,720 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:33:35,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:33:35,799 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['239375 memcached\n'], []) 2021-12-16 04:33:35,801 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:33:35,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:33:35,947 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['239375 memcached\n'], []) 2021-12-16 04:33:35,947 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:33:35,950 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:33:36,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:36,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:33:36,444 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:36,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:33:36,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:33:36,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:33:36,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:33:36,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:33:36,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:33:36,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:33:36,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:33:36,586 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:33:36,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:33:36,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:36,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:36,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:36,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:36,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:36,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:36,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:36,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:36,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:36,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:36,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:36,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:36,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:33:36,963 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:33:36,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:33:36,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:36,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:33:37,000 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:33:37,000 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:33:37,003 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:33:37,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:37,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:33:37,471 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:37,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:33:37,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:33:37,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:33:37,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:33:37,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:33:37,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:33:37,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:33:37,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:33:37,622 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:33:37,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:33:37,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:37,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:37,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:37,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:37,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:37,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:37,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:37,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:37,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:38,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:38,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:38,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:38,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:38,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:38,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:38,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:38,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:38,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:38,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:33:38,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:33:38,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:33:38,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:33:38,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:38,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:38,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:33:38,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:33:38,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:33:39,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:33:39,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:33:39,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:33:39,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:33:39,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:33:39,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:33:39,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:33:39,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:33:39,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:33:39,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:33:39,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:33:39,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:33:39,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:33:39,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:39,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:33:39,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:33:39,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:33:39,716 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:33:39,716 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:33:39,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:39,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:33:39,767 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:33:39,769 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:33:39,770 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:33:39,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:40,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:33:40,283 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:40,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:33:40,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:33:40,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:33:40,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:33:40,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:33:40,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:33:40,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:33:40,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:33:40,440 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:33:40,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:33:40,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:40,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:40,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:40,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:40,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:40,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:40,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:40,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:40,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:40,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:40,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:40,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:40,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:40,923 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:40,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:40,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:40,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:41,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:41,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:41,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:41,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:41,173 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:41,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:33:41,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:33:41,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:33:41,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:33:41,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,369 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:33:41,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:33:41,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:33:41,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:33:41,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:33:41,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:33:41,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:33:41,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:33:41,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:33:41,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:33:41,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:33:41,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:33:41,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:41,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:41,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:33:41,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:33:41,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:42,005 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:42,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:33:42,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:33:42,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:42,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:42,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:33:42,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:33:42,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:33:42,141 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:33:42,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:33:42,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:42,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:33:42,180 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:33:42,180 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:33:42,184 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:33:42,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:42,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:33:42,674 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:42,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:33:42,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:33:42,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:33:42,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:33:42,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:33:42,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:33:42,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:33:42,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:33:42,834 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:33:42,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:33:42,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:42,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:42,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:43,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:33:43,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:43,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:33:43,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:43,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:33:43,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:43,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:33:43,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:43,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:33:43,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:43,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:33:43,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:33:43,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:43,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:33:43,859 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:33:43,882 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:33:43,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:33:43,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:33:43,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:33:43,953 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:33:43,953 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:33:43,953 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:33:43,957 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:33:44,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:33:44,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:33:44,457 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:33:44,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:33:44,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:33:44,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:33:44,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:33:44,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:33:44,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:33:44,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:33:44,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:44,688 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:44,696 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:44,696 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:46,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:46,792 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:46,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:46,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:48,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:48,979 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:48,982 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:48,982 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:50,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:51,145 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:51,150 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:51,150 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:53,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:53,335 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:53,339 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:53,339 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:55,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:55,503 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:55,507 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:55,507 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:57,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:57,690 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:33:57,694 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:57,694 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:33:59,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:33:59,878 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:33:59,878 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:01,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:02,033 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:02,035 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:04,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:04,187 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:04,187 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:06,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:06,338 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:06,338 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:08,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:08,486 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:08,486 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:10,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:10,642 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:10,642 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:12,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:12,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:12,798 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:14,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:14,938 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:14,940 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:16,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:17,082 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:17,082 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:19,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:19,232 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:19,233 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:21,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:21,397 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:21,397 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:23,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:23,553 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:23,553 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:25,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:25,707 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:25,707 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:27,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:27,861 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:27,861 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:29,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:30,023 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:30,023 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:32,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:32,177 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:32,177 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:34,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:34,342 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:34,342 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:36,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:36,493 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:36,493 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:38,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:38,648 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:38,650 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:40,650 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:40,801 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:40,801 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:42,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:42,953 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:42,953 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:44,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:45,114 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:45,114 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:47,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:47,275 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:47,275 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:49,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:50,312 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:50,312 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:52,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:52,476 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:52,476 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:54,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:54,647 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:54,648 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:56,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:56,819 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:56,819 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:34:58,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:34:59,000 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:34:59,000 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:01,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:01,167 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:01,167 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:03,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:03,332 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:03,334 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:05,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:05,496 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:05,496 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:07,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:07,711 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:07,711 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:09,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:09,884 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:09,884 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:11,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:12,051 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:35:12,052 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:35:14,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:14,232 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:35:14,233 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.2790000439. 2021-12-16 04:35:14,233 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:35:14,236 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:35:14,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:35:14,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:35:14,724 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:35:14,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:35:14,744 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:35:14,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:35:14,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:35:14,803 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:35:14,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:35:14,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:35:14,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:15,066 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:35:15,066 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832999944687. 2021-12-16 04:35:15,068 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:35:15,071 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:35:15,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:35:15,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:35:15,598 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:35:15,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:35:15,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:35:15,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:35:15,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:35:15,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:35:15,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:35:15,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:35:15,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:15,943 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:35:15,944 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.87700009346. 2021-12-16 04:35:15,944 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:35:15,947 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:35:16,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:35:16,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:35:16,878 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:35:16,880 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:35:16,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:35:16,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:35:16,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:35:16,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:35:16,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:35:16,959 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:35:16,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:35:17,178 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:35:17,178 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.2349998951. ############################################################ # # RAM FootPrint: 4349.8203125 # ############################################################ 2021-12-16 04:35:17,209 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:35:17,209 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-16 04:37:03,213 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:37:03,213 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:37:08,216 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:37:08,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:08,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:37:08,914 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:08,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:37:09,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:37:09,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:37:09,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:37:09,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:37:09,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:37:09,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:37:09,191 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:37:09,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:37:09,282 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['197626 memcached\n'], []) 2021-12-16 04:37:09,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:37:09,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:37:09,499 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['197626 memcached\n'], []) 2021-12-16 04:37:09,499 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:37:09,502 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:37:09,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:09,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:37:10,025 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:10,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:37:10,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:37:10,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:37:10,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:37:10,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:37:10,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:37:10,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:37:10,180 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:37:10,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:37:10,256 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['165100 memcached\n'], []) 2021-12-16 04:37:10,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:37:10,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:37:10,382 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['165100 memcached\n'], []) 2021-12-16 04:37:10,384 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:37:10,387 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:37:10,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:10,803 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:37:10,901 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:10,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:37:10,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:37:10,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:37:10,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:37:10,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:37:11,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:37:11,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:37:11,052 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:37:11,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:37:11,134 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['39546 memcached\n'], []) 2021-12-16 04:37:11,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:37:11,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:37:11,260 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['39546 memcached\n'], []) 2021-12-16 04:37:11,262 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:37:11,263 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:37:11,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:11,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:37:11,721 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:11,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:37:11,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:37:11,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:37:11,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:37:11,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:37:11,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:37:11,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:37:11,880 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:37:11,880 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:37:11,957 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['240896 memcached\n'], []) 2021-12-16 04:37:11,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:37:12,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:37:12,095 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['240896 memcached\n'], []) 2021-12-16 04:37:12,095 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:37:12,098 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:37:12,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:12,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:37:12,546 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:12,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:37:12,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:37:12,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:37:12,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:37:12,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:37:12,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:37:12,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:37:12,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:37:12,694 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:37:12,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:37:12,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:12,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:12,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:12,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:12,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:12,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:12,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:12,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:12,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:13,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:13,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:13,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:13,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:37:13,069 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:37:13,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:37:13,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:13,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:37:13,115 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:37:13,115 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:37:13,119 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:37:13,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:13,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:37:13,566 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:13,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:37:13,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:37:13,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:37:13,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:37:13,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:37:13,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:37:13,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:37:13,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:37:13,749 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:37:13,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:37:13,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:13,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:13,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:13,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:13,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:14,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:14,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:14,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:14,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:14,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:14,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:14,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:14,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:14,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:14,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,604 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:14,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:14,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:37:14,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:37:14,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:37:14,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:37:14,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:14,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:37:14,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:37:14,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:14,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:37:15,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:37:15,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:37:15,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:37:15,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:37:15,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:37:15,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:37:15,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:37:15,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:37:15,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:37:15,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:37:15,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:37:15,716 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:37:15,808 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:37:15,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:37:15,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:37:15,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:15,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:15,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:37:15,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:37:15,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:37:15,994 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:37:15,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:37:16,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:16,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:37:16,035 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:37:16,035 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:37:16,038 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:37:16,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:16,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:37:16,539 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:16,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:37:16,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:37:16,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:37:16,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:37:16,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:37:16,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:37:16,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:37:16,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:37:16,680 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:37:16,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:37:16,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:16,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:16,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:16,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:16,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:16,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:16,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:17,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:17,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:17,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:17,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:17,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:17,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:17,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:17,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:17,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:17,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:17,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:17,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:37:17,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:37:17,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:37:17,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:37:17,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:37:17,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:37:17,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:37:17,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:37:17,808 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:37:17,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:37:17,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:37:17,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:37:17,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:17,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:17,997 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:37:18,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:37:18,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:18,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:18,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:37:18,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:37:18,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:18,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:18,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:37:18,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:37:18,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:18,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:18,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:37:18,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:37:18,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:18,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:18,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:37:18,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:37:18,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:37:18,319 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:37:18,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:37:18,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:18,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:37:18,355 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:37:18,355 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:37:18,357 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:37:18,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:18,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:37:18,825 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:18,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:37:18,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:37:18,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:37:18,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:37:18,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:37:18,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:37:18,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:37:18,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:37:18,974 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:37:18,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:37:19,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:19,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:37:19,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:19,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:37:19,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:19,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:37:19,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:19,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:37:19,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:19,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:37:19,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:19,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:37:19,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:37:19,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:19,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:37:19,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:37:19,904 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:37:19,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:37:19,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:37:19,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:37:19,964 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:37:19,964 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:37:19,964 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:37:19,970 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:37:20,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:37:20,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:37:20,456 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:37:20,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:37:20,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:37:20,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:37:20,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:37:20,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:37:20,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:37:20,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:37:20,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:20,727 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:20,734 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:20,734 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:22,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:22,913 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:22,917 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:22,917 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:24,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:25,079 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:25,084 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:25,085 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:27,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:27,242 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:27,247 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:27,247 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:29,249 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:29,421 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:29,426 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:29,426 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:31,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:31,608 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:31,612 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:31,614 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:33,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:33,782 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:37:33,786 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:33,786 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:35,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:35,950 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:35,950 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:37,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:38,095 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:38,096 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:40,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:40,246 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:40,246 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:42,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:42,391 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:42,391 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:44,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:44,543 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:44,543 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:46,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:46,690 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:46,690 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:48,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:48,836 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:48,836 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:50,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:50,980 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:50,980 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:52,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:53,140 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:53,140 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:55,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:55,279 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:55,279 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:57,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:57,438 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:57,440 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:37:59,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:37:59,588 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:37:59,589 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:01,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:01,746 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:01,746 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:03,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:03,907 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:03,907 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:05,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:06,061 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:06,061 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:08,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:08,210 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:08,211 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:10,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:10,367 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:10,367 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:12,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:12,516 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:12,516 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:14,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:14,665 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:14,665 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:16,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:16,812 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:16,812 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:18,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:18,971 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:18,971 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:20,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:21,138 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:21,138 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:23,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:23,608 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:23,608 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:25,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:25,778 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:25,778 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:27,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:27,947 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:27,947 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:29,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:30,118 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:30,118 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:32,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:32,286 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:32,288 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:34,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:34,457 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:34,457 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:36,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:36,630 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:36,630 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:38,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:38,799 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:38,799 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:40,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:40,973 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:40,973 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:42,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:43,155 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:43,155 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:45,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:45,325 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:38:45,325 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:38:47,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:47,569 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:38:47,569 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.6040000916. 2021-12-16 04:38:47,569 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:38:47,572 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:38:47,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:38:47,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:38:48,052 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:38:48,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:38:48,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:38:48,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:38:48,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:38:48,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:38:48,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:38:48,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:38:48,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:48,338 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:38:48,338 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.769000053406. 2021-12-16 04:38:48,339 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:38:48,342 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:38:48,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:38:48,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:38:48,865 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:38:48,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:38:48,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:38:48,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:38:48,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:38:48,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:38:48,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:38:48,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:38:49,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:49,243 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:38:49,243 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.904000043869. 2021-12-16 04:38:49,244 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:38:49,246 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:38:49,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:38:49,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:38:50,473 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:38:50,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:38:50,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:38:50,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:38:50,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:38:50,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:38:50,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:38:50,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:38:50,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:38:50,996 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:38:50,996 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.75100016594. ############################################################ # # RAM FootPrint: 4297.71875 # ############################################################ 2021-12-16 04:38:51,026 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:38:51,028 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-16 04:39:57,032 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:39:57,032 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:40:02,036 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:40:02,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:02,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:40:02,737 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:02,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:40:02,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:40:02,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:40:02,808 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:40:03,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:40:03,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:40:03,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:40:03,559 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:40:03,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:40:03,628 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['199379 memcached\n'], []) 2021-12-16 04:40:03,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:40:03,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:40:03,934 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['199379 memcached\n'], []) 2021-12-16 04:40:03,936 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:40:03,940 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:40:04,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:04,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:40:04,457 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:04,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:40:04,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:40:04,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:40:04,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:40:04,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:40:04,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:40:04,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:40:04,609 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:40:04,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:40:04,680 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['166038 memcached\n'], []) 2021-12-16 04:40:04,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:40:04,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:40:04,828 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['166038 memcached\n'], []) 2021-12-16 04:40:04,829 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:40:04,832 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:40:04,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:05,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:40:05,334 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:05,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:40:05,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:40:05,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:40:05,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:40:05,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:40:05,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:40:05,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:40:05,486 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:40:05,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:40:05,559 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['40368 memcached\n'], []) 2021-12-16 04:40:05,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:40:05,650 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:40:05,719 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['40368 memcached\n'], []) 2021-12-16 04:40:05,720 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:40:05,723 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:40:05,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:06,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:40:06,194 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:06,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:40:06,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:40:06,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:40:06,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:40:06,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:40:06,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:40:06,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:40:06,361 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:40:06,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:40:06,434 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['242431 memcached\n'], []) 2021-12-16 04:40:06,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:40:06,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:40:06,563 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['242431 memcached\n'], []) 2021-12-16 04:40:06,563 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:40:06,566 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:40:06,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:07,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:40:08,052 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:08,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:40:08,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:40:08,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:40:08,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:40:08,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:40:08,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:40:08,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:40:08,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:40:08,204 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:40:08,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:40:08,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:08,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:08,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:08,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:08,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:08,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:08,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:08,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:08,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:08,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:08,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:08,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:08,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:40:08,598 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:40:08,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:40:08,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:08,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:40:08,644 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:40:08,644 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:40:08,647 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:40:08,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:09,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:40:09,092 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:09,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:40:09,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:40:09,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:40:09,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:40:09,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:40:09,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:40:09,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:40:09,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:40:09,282 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:40:09,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:40:09,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:09,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:09,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:09,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:09,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:09,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:09,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:09,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:09,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:09,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:09,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:09,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:09,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:09,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:09,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:09,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:10,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:40:10,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:40:10,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:40:10,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:40:10,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:40:10,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:40:10,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:40:10,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:40:10,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:40:10,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:40:10,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:40:10,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:40:10,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:40:10,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:40:10,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:40:10,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:40:10,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:40:10,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:40:10,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:10,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:10,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:40:10,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:40:10,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:11,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:11,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:40:11,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:40:11,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:11,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:11,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:40:11,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:40:11,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:40:11,177 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:40:11,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:40:11,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:11,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:40:11,232 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:40:11,232 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:40:11,233 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:40:11,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:11,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:40:11,750 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:11,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:40:11,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:40:11,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:40:11,808 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:40:11,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:40:11,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:40:11,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:40:11,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:40:11,890 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:40:11,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:40:11,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:11,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:12,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:12,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:12,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:12,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:12,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:12,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:12,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:12,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:12,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:12,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:12,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:12,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:12,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:12,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:12,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:12,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:40:12,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:40:12,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:40:13,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:40:13,104 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:40:13,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:40:13,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:40:13,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:40:13,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:40:13,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:40:13,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:40:13,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:40:13,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:40:13,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:40:13,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:40:13,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:40:13,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:40:13,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:40:13,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:13,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:13,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:40:13,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:40:14,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:14,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:14,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:40:14,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:40:14,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:40:14,125 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:40:14,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:40:14,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:14,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:40:14,178 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:40:14,178 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:40:14,183 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:40:14,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:14,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:40:14,691 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:14,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:40:14,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:40:14,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:40:14,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:40:14,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:40:14,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:40:14,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:40:14,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:40:14,865 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:40:14,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:40:14,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:14,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:14,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:15,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:40:15,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:15,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:40:15,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:15,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:40:15,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:15,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:40:15,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:15,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:40:15,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:15,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:40:15,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:40:15,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:15,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:40:15,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:40:15,851 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:40:15,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:40:15,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:40:15,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:40:15,901 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:40:15,901 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:40:15,901 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:40:15,904 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:40:16,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:40:16,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:40:16,421 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:40:16,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:40:16,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:40:16,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:40:16,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:40:16,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:40:16,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:40:16,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:40:16,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:16,706 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:16,710 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:16,710 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:18,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:18,875 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:18,878 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:18,878 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:20,880 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:21,059 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:21,062 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:21,063 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:23,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:23,246 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:23,250 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:23,250 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:25,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:25,427 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:25,433 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:25,433 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:27,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:27,622 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:27,627 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:27,627 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:29,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:29,729 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:40:29,733 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:29,733 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:31,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:31,887 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:31,887 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:33,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:34,046 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:34,046 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:36,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:36,193 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:36,194 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:38,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:38,339 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:38,339 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:40,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:40,490 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:40,490 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:42,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:42,644 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:42,644 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:44,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:44,812 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:44,812 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:46,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:46,953 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:46,953 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:48,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:49,118 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:49,118 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:51,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:51,276 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:51,276 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:53,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:53,430 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:53,430 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:55,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:55,578 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:55,578 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:57,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:57,733 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:57,733 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:40:59,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:40:59,894 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:40:59,894 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:01,895 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:02,042 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:02,042 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:04,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:04,206 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:04,207 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:06,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:06,348 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:06,349 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:08,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:08,500 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:08,500 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:10,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:10,657 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:10,657 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:12,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:12,812 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:12,812 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:14,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:14,963 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:14,963 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:16,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:17,115 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:17,115 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:19,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:19,674 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:19,676 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:21,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:21,920 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:21,920 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:23,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:24,108 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:24,108 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:26,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:26,345 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:26,345 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:28,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:28,526 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:28,526 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:30,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:30,698 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:30,698 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:32,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:32,878 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:32,878 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:34,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:35,048 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:35,048 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:37,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:37,302 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:37,302 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:39,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:39,484 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:39,486 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:41,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:41,667 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:41:41,667 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:41:43,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:43,895 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:41:43,895 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.993999958. 2021-12-16 04:41:43,897 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:41:43,898 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:41:44,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:41:44,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:41:44,405 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:41:44,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:41:44,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:41:44,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:41:44,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:41:44,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:41:44,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:41:44,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:41:44,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:44,714 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:41:44,714 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.817999839783. 2021-12-16 04:41:44,714 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:41:44,717 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:41:44,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:41:45,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:41:45,266 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:41:45,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:41:45,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:41:45,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:41:45,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:41:45,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:41:45,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:41:45,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:41:45,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:45,609 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:41:45,611 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.896000146866. 2021-12-16 04:41:45,611 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:41:45,612 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:41:45,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:41:46,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:41:46,193 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:41:46,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:41:46,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:41:46,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:41:46,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:41:46,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:41:46,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:41:46,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:41:46,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:41:46,667 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:41:46,667 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.05599999428. ############################################################ # # RAM FootPrint: 4256.82421875 # ############################################################ 2021-12-16 04:41:46,697 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:41:46,697 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-16 04:43:00,703 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:43:00,703 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:43:05,704 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:43:05,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:06,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:43:06,605 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:06,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:43:06,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:43:06,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:43:06,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:43:06,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:43:06,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:43:06,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:43:06,885 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:43:06,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:43:06,989 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['200917 memcached\n'], []) 2021-12-16 04:43:06,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:43:07,084 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:43:07,161 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['200917 memcached\n'], []) 2021-12-16 04:43:07,161 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:43:07,167 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:43:07,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:07,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:43:07,654 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:07,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:43:07,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:43:07,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:43:07,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:43:07,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:43:07,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:43:07,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:43:07,812 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:43:07,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:43:07,897 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['166950 memcached\n'], []) 2021-12-16 04:43:07,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:43:07,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:43:08,039 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['166950 memcached\n'], []) 2021-12-16 04:43:08,039 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:43:08,043 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:43:08,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:08,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:43:08,545 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:08,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:43:08,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:43:08,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:43:08,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:43:08,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:43:08,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:43:08,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:43:08,700 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:43:08,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:43:08,775 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['41175 memcached\n'], []) 2021-12-16 04:43:08,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:43:08,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:43:08,881 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['41175 memcached\n'], []) 2021-12-16 04:43:08,881 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:43:08,884 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:43:09,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:09,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:43:09,348 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:09,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:43:09,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:43:09,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:43:09,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:43:09,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:43:09,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:43:09,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:43:09,489 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:43:09,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:43:09,542 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['243941 memcached\n'], []) 2021-12-16 04:43:09,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:43:09,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:43:09,691 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['243941 memcached\n'], []) 2021-12-16 04:43:09,691 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:43:09,694 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:43:09,836 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:15,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:43:15,994 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:15,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:43:16,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:43:16,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:43:16,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:43:16,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:43:16,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:43:16,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:43:16,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:43:16,132 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:43:16,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:43:16,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:16,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:16,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:16,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:16,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:16,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:16,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:16,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:16,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:16,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:16,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:16,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:16,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:43:16,525 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:43:16,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:43:16,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:16,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:43:16,569 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:43:16,571 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:43:16,573 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:43:16,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:16,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:43:17,025 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:17,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:43:17,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:43:17,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:43:17,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:43:17,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:43:17,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:43:17,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:43:17,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:43:17,178 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:43:17,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:43:17,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:17,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:17,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:17,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:17,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:17,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:17,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:17,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:17,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:17,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:17,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:17,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:17,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:17,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:17,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:17,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:18,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:18,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:18,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:43:18,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:43:18,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:43:18,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:43:18,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:43:18,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:43:18,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:43:18,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:43:18,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:43:18,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:43:18,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:43:18,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:43:18,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:43:18,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:43:18,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:43:18,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:43:18,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:18,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:18,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:43:18,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:43:18,984 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:19,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:19,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:43:19,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:43:19,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:19,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:19,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:43:19,173 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:43:19,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:43:19,213 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:43:19,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:43:19,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:19,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:43:19,260 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:43:19,260 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:43:19,265 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:43:19,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:19,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:43:19,755 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:19,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:43:19,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:43:19,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:43:19,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:43:19,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:43:19,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:43:19,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:43:19,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:43:19,875 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:43:19,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:43:19,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:19,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:20,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:20,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,181 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:20,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:20,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:20,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:20,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,443 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:20,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:20,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:20,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:20,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:20,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:20,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:20,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:20,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:20,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:20,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:20,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:21,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:43:21,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:43:21,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,219 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:43:21,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:43:21,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:43:21,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:43:21,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:43:21,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:43:21,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:43:21,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:43:21,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:43:21,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:43:21,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:21,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:21,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:43:21,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:43:21,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:22,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:22,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:43:22,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:43:22,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:22,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:22,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:43:22,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:43:22,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:22,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:22,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:43:22,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:43:22,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:43:22,374 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:43:22,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:43:22,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:22,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:43:22,424 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:43:22,424 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:43:22,426 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:43:22,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:22,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:43:22,825 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:22,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:43:22,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:43:22,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:43:22,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:43:22,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:43:22,923 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:43:22,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:43:22,959 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:43:22,980 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:43:23,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:43:23,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:23,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:43:23,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:23,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:43:23,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:23,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:43:23,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:23,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:43:23,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:23,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:43:23,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:23,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:43:23,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:23,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:43:23,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:23,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:43:23,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:23,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:43:23,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:43:24,022 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:43:24,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:43:24,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:43:24,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:43:24,095 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:43:24,095 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:43:24,096 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:43:24,098 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:43:24,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:43:24,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:43:24,605 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:43:24,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:43:24,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:43:24,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:43:24,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:43:24,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:43:24,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:43:24,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:43:24,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:24,904 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:43:24,908 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:24,908 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:26,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:27,071 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:43:27,075 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:27,075 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:29,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:29,183 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:43:29,187 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:29,187 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:31,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:31,335 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:31,335 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:33,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:33,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:33,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:35,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:35,632 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:35,634 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:37,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:37,780 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:37,782 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:39,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:39,927 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:39,927 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:41,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:42,075 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:42,075 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:44,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:44,220 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:44,221 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:46,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:46,390 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:46,390 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:48,390 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:48,542 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:48,542 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:50,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:50,693 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:50,693 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:52,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:52,842 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:52,842 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:54,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:54,993 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:54,993 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:56,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:57,153 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:57,154 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:43:59,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:43:59,305 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:43:59,305 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:01,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:01,464 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:01,464 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:03,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:03,618 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:03,619 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:05,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:05,776 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:05,776 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:07,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:07,934 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:07,934 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:09,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:10,092 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:10,092 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:12,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:12,246 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:12,246 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:14,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:14,414 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:14,414 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:16,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:16,562 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:16,562 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:18,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:18,717 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:18,719 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:20,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:20,881 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:20,882 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:22,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:23,042 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:23,042 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:25,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:25,201 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:25,201 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:27,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:27,355 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:27,357 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:29,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:29,509 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:29,509 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:31,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:31,661 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:31,661 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:33,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:34,335 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:34,335 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:36,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:36,720 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:36,720 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:38,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:38,898 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:38,898 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:40,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:41,066 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:44:41,068 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:44:43,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:44,023 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:44:44,023 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 79.9270000458. 2021-12-16 04:44:44,023 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:44:44,026 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:44:44,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:44:44,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:44:44,532 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:44:44,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:44:44,553 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:44:44,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:44:44,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:44:44,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:44:44,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:44:44,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:44:44,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:44,832 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:44:44,832 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.809000015259. 2021-12-16 04:44:44,834 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:44:44,835 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:44:44,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:44:45,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:44:45,335 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:44:45,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:44:45,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:44:45,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:44:45,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:44:45,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:44:45,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:44:45,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:44:45,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:45,651 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:44:45,651 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.817999839783. 2021-12-16 04:44:45,651 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:44:45,654 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:44:45,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:44:46,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:44:47,365 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:44:47,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:44:47,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:44:47,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:44:47,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:44:47,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:44:47,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:44:47,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:44:47,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:44:48,142 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:44:48,144 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 2.492000103. ############################################################ # # RAM FootPrint: 4239.2890625 # ############################################################ 2021-12-16 04:44:48,176 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:44:48,176 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 99 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-16 04:46:27,180 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:46:27,180 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:46:32,183 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:46:32,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:32,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:46:32,924 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:32,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:46:32,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:46:32,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:46:32,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:46:33,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:46:33,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:46:33,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:46:33,463 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:46:33,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:46:33,551 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['202185 memcached\n'], []) 2021-12-16 04:46:33,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:46:34,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:46:34,298 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['202185 memcached\n'], []) 2021-12-16 04:46:34,298 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:46:34,301 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:46:34,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:34,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:46:34,821 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:34,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:46:34,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:46:34,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:46:34,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:46:34,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:46:34,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:46:34,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:46:34,971 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:46:34,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:46:35,052 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['167868 memcached\n'], []) 2021-12-16 04:46:35,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:46:35,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:46:35,158 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['167868 memcached\n'], []) 2021-12-16 04:46:35,158 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:46:35,163 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:46:35,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:35,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:46:35,673 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:35,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:46:35,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:46:35,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:46:35,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:46:35,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:46:35,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:46:35,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:46:35,796 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:46:35,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:46:35,851 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['41970 memcached\n'], []) 2021-12-16 04:46:35,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:46:35,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:46:35,973 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['41970 memcached\n'], []) 2021-12-16 04:46:35,973 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:46:35,974 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:46:36,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:36,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:46:36,434 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:36,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:46:36,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:46:36,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:46:36,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:46:36,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:46:36,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:46:36,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:46:36,602 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:46:36,602 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:46:36,680 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['245405 memcached\n'], []) 2021-12-16 04:46:36,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:46:36,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:46:36,822 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['245405 memcached\n'], []) 2021-12-16 04:46:36,823 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:46:36,826 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:46:36,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:39,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:46:39,325 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:39,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:46:39,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:46:39,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:46:39,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:46:39,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:46:39,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:46:39,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:46:39,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:46:39,450 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:46:39,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:46:39,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:39,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:39,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:39,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:39,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:39,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:39,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:39,716 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:39,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:39,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:39,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:39,790 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:39,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:46:39,819 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:46:39,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:46:39,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:39,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:46:39,859 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:46:39,859 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:46:39,862 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:46:39,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:40,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:46:40,323 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:40,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:46:40,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:46:40,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:46:40,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:46:40,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:46:40,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:46:40,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:46:40,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:46:40,476 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:46:40,502 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:46:40,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:40,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:40,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:40,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:40,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:40,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:40,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:40,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:40,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:40,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:40,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:41,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:41,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:41,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:41,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:41,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:41,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:41,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:41,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:41,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:41,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:41,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:41,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:46:41,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:46:41,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:46:41,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:46:41,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:46:41,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:46:41,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:41,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:41,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:46:42,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:46:42,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:46:42,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:46:42,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:46:42,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:46:42,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:46:42,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:46:42,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:46:42,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:46:42,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:46:42,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:46:42,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:46:42,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:46:42,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:42,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:46:42,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:46:42,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:46:42,759 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:46:42,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:46:42,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:42,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:46:42,808 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:46:42,809 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:46:42,811 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:46:42,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:43,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:46:43,313 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:43,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:46:43,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:46:43,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:46:43,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:46:43,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:46:43,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:46:43,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:46:43,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:46:43,467 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:46:43,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:46:43,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:43,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:43,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:43,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:43,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:43,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:43,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:43,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:43,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:43,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:43,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:43,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:44,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:44,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:44,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:44,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:44,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:44,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:44,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:44,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:44,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:44,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:44,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:46:44,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:46:44,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:46:44,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:46:44,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:44,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:46:44,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:46:44,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:44,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:46:45,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:46:45,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:46:45,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:46:45,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:46:45,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:46:45,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:46:45,329 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:46:45,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:46:45,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:46:45,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:46:45,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:46:45,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:45,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:46:45,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:46:45,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:46:45,822 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:46:45,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:46:45,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:45,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:46:45,884 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:46:45,885 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:46:45,890 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:46:45,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:46,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:46:46,357 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:46,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:46:46,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:46:46,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:46:46,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:46:46,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:46:46,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:46:46,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:46:46,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:46:46,502 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:46:46,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:46:46,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:46,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:46,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:46,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:46:46,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:46,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:46,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:46,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:46:46,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:46,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:46,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:46,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:46:46,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:47,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:47,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:46:47,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:47,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:47,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:46:47,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,329 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:47,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:47,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:46:47,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:47,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:47,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:46:47,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:46:47,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:47,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:46:47,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:46:47,693 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:46:47,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:46:47,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:46:47,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:46:47,749 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:46:47,749 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:46:47,749 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:46:47,753 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:46:47,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:46:48,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:46:48,265 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:46:48,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:46:48,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:46:48,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:46:48,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:46:48,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:46:48,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:46:48,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:46:48,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:48,634 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:48,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:48,641 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:46:50,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:50,826 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:50,832 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:50,832 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:46:52,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:52,993 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:52,999 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:53,000 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:46:55,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:55,174 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:55,177 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:55,177 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:46:57,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:57,355 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:57,358 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:57,358 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:46:59,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:46:59,536 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:46:59,540 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:46:59,540 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:01,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:01,707 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:47:01,710 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:01,711 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:03,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:03,859 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:03,859 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:05,859 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:06,000 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:06,002 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:08,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:08,167 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:08,168 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:10,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:10,319 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:10,319 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:12,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:12,493 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:12,494 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:14,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:14,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:14,641 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:16,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:16,788 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:16,788 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:18,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:18,944 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:18,944 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:20,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:21,089 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:21,091 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:23,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:23,252 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:23,253 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:25,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:25,407 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:25,408 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:27,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:27,566 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:27,566 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:29,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:29,729 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:29,729 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:31,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:31,888 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:31,888 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:33,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:34,052 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:34,052 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:36,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:36,210 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:36,210 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:38,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:38,381 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:38,382 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:40,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:40,536 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:40,536 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:42,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:42,694 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:42,696 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:44,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:44,842 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:44,842 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:46,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:47,000 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:47,002 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:49,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:49,163 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:49,163 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:51,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:51,312 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:51,312 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:53,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:53,480 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:53,480 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:55,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:56,332 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:56,332 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:47:58,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:47:58,591 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:47:58,591 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:00,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:00,766 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:00,766 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:02,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:02,944 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:02,944 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:04,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:05,091 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:05,091 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:07,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:07,255 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:07,256 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:09,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:09,427 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:09,427 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:11,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:11,637 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:48:11,637 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:48:13,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:14,572 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:48:14,572 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 86.8239998817. 2021-12-16 04:48:14,573 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:48:14,575 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:48:14,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:48:14,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:48:15,085 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:48:15,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:48:15,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:48:15,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:48:15,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:48:15,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:48:15,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:48:15,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:48:15,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:15,443 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:48:15,443 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.869000196457. 2021-12-16 04:48:15,444 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:48:15,447 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:48:15,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:48:15,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:48:15,948 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:48:15,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:48:15,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:48:16,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:48:16,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:48:16,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:48:16,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:48:16,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:48:16,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:16,335 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:48:16,335 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.891000032425. 2021-12-16 04:48:16,336 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:48:16,338 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:48:16,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:48:16,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:48:16,726 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:48:16,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:48:16,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:48:16,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:48:16,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:48:16,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:48:16,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:48:16,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:48:16,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:48:17,053 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:48:17,055 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.717000007629. ############################################################ # # RAM FootPrint: 4187.36328125 # ############################################################ 2021-12-16 04:48:17,085 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:48:17,085 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 85 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-16 04:49:42,089 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:49:42,091 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:49:47,094 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:49:47,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:48,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:49:48,584 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:48,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:49:48,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:49:48,622 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:49:48,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:49:48,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:49:49,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:49:49,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:49:49,049 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:49:49,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:49:49,132 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['204017 memcached\n'], []) 2021-12-16 04:49:49,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:49:49,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:49:49,540 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['204017 memcached\n'], []) 2021-12-16 04:49:49,542 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:49:49,545 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:49:49,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:49,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:49:50,046 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:50,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:49:50,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:49:50,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:49:50,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:49:50,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:49:50,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:49:50,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:49:50,191 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:49:50,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:49:50,273 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['168849 memcached\n'], []) 2021-12-16 04:49:50,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:49:50,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:49:50,404 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['168849 memcached\n'], []) 2021-12-16 04:49:50,404 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:49:50,407 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:49:50,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:50,821 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:49:50,910 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:50,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:49:50,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:49:50,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:49:50,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:49:50,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:49:51,005 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:49:51,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:49:51,042 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:49:51,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:49:51,115 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['42812 memcached\n'], []) 2021-12-16 04:49:51,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:49:51,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:49:51,219 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['42812 memcached\n'], []) 2021-12-16 04:49:51,219 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:49:51,223 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:49:51,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:51,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:49:51,674 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:51,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:49:51,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:49:51,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:49:51,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:49:51,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:49:51,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:49:51,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:49:51,858 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:49:51,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:49:51,944 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['246975 memcached\n'], []) 2021-12-16 04:49:51,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:49:52,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:49:52,065 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['246975 memcached\n'], []) 2021-12-16 04:49:52,066 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:49:52,069 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:49:52,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:53,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:49:53,387 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:53,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:49:53,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:49:53,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:49:53,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:49:53,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:49:53,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:49:53,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:49:53,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:49:53,538 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:49:53,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:49:53,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:53,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:53,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:53,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:53,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:53,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:53,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:53,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:53,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:53,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:53,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:53,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:53,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:49:53,957 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:49:53,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:49:53,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:53,984 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:49:53,997 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:49:53,997 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:49:54,000 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:49:54,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:54,354 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:49:54,440 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:54,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:49:54,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:49:54,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:49:54,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:49:54,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:49:54,535 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:49:54,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:49:54,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:49:54,581 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:49:54,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:49:54,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:54,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:54,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:54,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:54,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:54,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:54,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:54,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:54,923 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:54,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:54,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:55,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:55,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:49:55,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:49:55,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:49:55,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:49:55,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:49:55,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:49:55,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:49:55,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:49:55,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:49:55,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:49:55,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:49:55,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:49:55,505 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:49:55,566 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:49:55,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,602 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:49:55,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:49:55,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:49:55,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:49:55,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:49:55,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:49:55,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:49:55,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:49:55,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:55,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:49:55,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:49:55,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:55,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:56,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:49:56,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:49:56,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:56,056 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:56,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:49:56,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:49:56,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:56,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:56,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:49:56,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:49:56,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:56,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:56,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:49:56,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:49:56,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:49:56,282 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:49:56,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:49:56,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:56,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:49:56,325 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:49:56,325 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:49:56,328 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:49:56,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:49:56,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:49:56,841 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:49:56,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:49:56,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:49:56,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:49:56,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:49:56,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:49:56,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:49:56,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:49:56,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:49:56,983 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:49:57,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:49:57,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:57,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:49:57,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:57,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:49:57,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:57,556 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:49:57,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:49:57,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:49:57,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:49:57,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:49:57,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:57,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:57,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:49:57,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:49:58,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:49:58,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:49:58,122 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:49:58,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:49:58,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:49:58,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:49:58,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:49:58,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:49:58,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:49:58,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:49:58,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:49:58,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:49:58,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:58,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:58,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:49:58,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:49:58,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:59,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:49:59,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:49:59,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:59,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:49:59,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:49:59,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:59,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:49:59,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:49:59,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:59,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:49:59,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:49:59,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:49:59,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:49:59,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:49:59,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:49:59,704 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:49:59,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:49:59,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:49:59,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:49:59,750 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:49:59,750 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:49:59,755 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:49:59,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:50:00,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:50:00,262 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:50:00,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:50:00,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:50:00,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:50:00,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:50:00,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:50:00,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:50:00,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:50:00,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:50:00,411 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:50:00,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:50:00,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:00,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:00,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:50:00,604 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:50:00,632 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:00,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:00,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:50:00,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:50:00,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:00,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:00,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:50:00,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:50:00,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:00,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:00,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:50:00,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:50:00,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:01,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:01,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:50:01,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:50:01,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:01,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:01,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:50:01,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:50:01,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:01,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:01,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:50:01,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:50:01,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:01,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:50:01,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:50:01,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:50:01,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:50:01,484 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:50:01,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:50:01,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:50:01,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:50:01,529 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:50:01,530 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:50:01,530 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:50:01,533 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:50:01,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:50:01,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:50:02,052 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:50:02,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:50:02,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:50:02,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:50:02,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:50:02,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:50:02,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:50:02,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:50:02,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:02,372 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:02,378 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:02,378 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:04,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:04,555 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:04,561 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:04,562 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:06,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:06,723 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:06,727 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:06,727 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:08,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:08,910 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:08,914 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:08,914 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:10,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:11,098 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:11,105 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:11,105 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:13,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:13,273 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:13,278 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:13,278 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:15,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:15,447 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:15,451 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:15,451 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:17,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:17,611 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:17,617 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:17,617 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:19,617 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:19,783 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:19,788 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:19,788 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:21,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:21,956 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:21,960 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:21,960 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:23,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:24,134 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:24,138 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:24,140 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:26,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:26,308 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:50:26,312 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:26,312 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:28,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:28,453 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:28,453 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:30,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:30,601 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:30,601 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:32,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:32,749 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:32,749 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:34,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:34,894 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:34,894 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:36,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:37,052 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:37,052 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:39,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:39,196 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:39,196 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:41,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:41,348 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:41,348 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:43,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:43,494 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:43,494 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:45,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:45,654 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:45,654 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:47,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:47,808 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:47,808 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:49,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:49,956 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:49,956 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:51,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:52,111 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:52,111 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:54,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:54,263 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:54,263 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:56,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:56,418 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:56,420 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:50:58,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:50:58,578 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:50:58,578 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:00,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:00,726 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:00,726 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:02,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:02,881 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:02,881 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:04,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:05,042 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:05,043 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:07,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:07,197 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:07,197 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:09,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:09,348 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:09,348 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:11,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:11,509 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:11,509 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:13,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:13,660 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:13,660 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:15,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:15,818 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:15,819 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:17,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:17,974 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:17,974 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:19,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:20,121 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:20,121 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:22,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:22,273 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:22,273 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:24,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:24,434 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:24,434 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:26,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:26,592 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:26,592 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:28,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:28,753 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:28,753 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:30,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:31,321 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:31,321 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:33,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:33,528 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:33,528 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:35,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:35,762 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:35,762 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:37,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:37,930 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:51:37,930 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:51:39,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:40,101 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:51:40,101 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.5710000992. 2021-12-16 04:51:40,102 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:51:40,104 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:51:40,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:51:40,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:51:40,619 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:51:40,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:51:40,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:51:40,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:51:40,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:51:40,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:51:40,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:51:40,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:51:40,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:40,943 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:51:40,943 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.840000152588. 2021-12-16 04:51:40,944 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:51:40,946 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:51:41,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:51:41,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:51:41,461 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:51:41,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:51:41,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:51:41,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:51:41,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:51:41,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:51:41,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:51:41,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:51:41,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:41,710 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:51:41,710 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.766999959946. 2021-12-16 04:51:41,711 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:51:41,713 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:51:41,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:51:42,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:51:42,187 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:51:42,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:51:42,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:51:42,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:51:42,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:51:42,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:51:42,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:51:42,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:51:42,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:51:42,591 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:51:42,591 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.87900018692. ############################################################ # # RAM FootPrint: 4145.42578125 # ############################################################ 2021-12-16 04:51:42,621 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:51:42,621 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-16 04:52:49,625 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:52:49,625 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:52:54,628 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:52:55,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:52:55,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:52:55,990 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:52:55,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:52:56,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:52:56,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:52:56,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:52:56,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:52:56,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:52:56,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:52:56,253 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:52:56,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:52:56,328 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['206211 memcached\n'], []) 2021-12-16 04:52:56,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:52:56,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:52:56,513 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['206211 memcached\n'], []) 2021-12-16 04:52:56,513 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:52:56,516 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:52:56,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:52:56,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:52:57,015 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:52:57,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:52:57,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:52:57,056 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:52:57,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:52:57,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:52:57,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:52:57,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:52:57,163 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:52:57,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:52:57,236 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['169810 memcached\n'], []) 2021-12-16 04:52:57,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:52:57,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:52:57,357 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['169810 memcached\n'], []) 2021-12-16 04:52:57,357 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:52:57,359 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:52:57,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:52:57,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:52:57,861 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:52:57,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:52:57,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:52:57,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:52:57,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:52:57,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:52:57,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:52:57,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:52:57,996 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:52:57,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:52:58,061 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['43629 memcached\n'], []) 2021-12-16 04:52:58,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:52:58,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:52:58,194 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['43629 memcached\n'], []) 2021-12-16 04:52:58,194 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:52:58,197 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:52:58,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:52:58,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:52:58,663 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:52:58,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:52:58,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:52:58,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:52:58,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:52:58,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:52:58,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:52:58,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:52:58,819 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:52:58,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:52:58,901 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['248505 memcached\n'], []) 2021-12-16 04:52:58,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:52:58,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:52:59,045 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['248505 memcached\n'], []) 2021-12-16 04:52:59,045 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:52:59,049 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:52:59,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:53:00,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:53:00,292 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:53:00,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:53:00,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:53:00,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:53:00,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:53:00,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:53:00,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:53:00,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:53:00,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:53:00,436 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:53:00,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:53:00,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:00,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:00,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:00,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:00,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:00,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:00,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:00,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:00,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:00,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:00,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:00,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:00,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:53:00,826 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:53:00,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:53:00,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:00,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:53:00,875 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:53:00,875 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:53:00,878 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:53:00,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:53:01,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:53:01,309 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:53:01,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:53:01,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:53:01,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:53:01,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:53:01,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:53:01,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:53:01,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:53:01,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:53:01,473 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:53:01,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:53:01,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:01,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:01,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:01,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:01,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:01,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:01,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:01,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:01,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:01,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:01,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:02,028 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:02,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:02,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:02,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:02,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:02,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:02,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:02,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:02,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:02,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:02,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:02,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:53:02,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:53:02,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:02,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:02,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:53:02,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:53:02,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:53:03,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:53:03,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:53:03,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:53:03,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:53:03,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:53:03,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:53:03,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:53:03,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:53:03,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:53:03,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:53:03,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:53:03,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:53:03,874 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:53:03,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:03,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:03,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:53:03,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:53:04,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:04,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:04,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:53:04,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:53:04,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:53:04,124 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:53:04,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:53:04,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:04,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:53:04,176 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:53:04,176 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:53:04,178 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:53:04,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:53:04,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:53:04,667 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:53:04,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:53:04,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:53:04,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:53:04,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:53:04,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:53:04,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:53:04,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:53:04,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:53:04,779 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:53:04,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:53:04,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:04,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:04,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:04,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:04,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:05,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:05,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:05,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:05,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:05,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:05,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:05,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:05,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:05,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:05,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:05,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:05,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:05,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:05,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:53:05,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:53:05,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:05,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:05,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:53:05,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:53:06,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:53:06,122 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:53:06,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:53:06,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:53:06,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:53:06,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:53:06,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:53:06,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:53:06,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:53:06,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:53:06,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:53:06,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:53:06,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:53:06,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:53:06,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:06,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:06,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:53:07,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:53:07,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:07,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:07,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:53:07,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:53:07,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:53:07,141 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:53:07,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:53:07,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:07,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:53:07,187 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:53:07,187 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:53:07,190 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:53:07,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:53:07,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:53:07,711 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:53:07,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:53:07,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:53:07,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:53:07,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:53:07,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:53:07,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:53:07,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:53:07,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:53:07,867 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:53:07,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:53:07,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:07,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:07,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:08,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:53:08,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:08,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:53:08,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:08,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:53:08,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:08,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:53:08,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:08,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:53:08,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:08,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:53:08,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:08,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:53:08,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:08,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:53:08,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:08,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:53:08,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:53:09,007 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:53:09,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:53:09,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:53:09,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:53:09,059 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:53:09,059 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:53:09,059 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:53:09,065 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:53:09,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:53:09,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:53:09,552 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:53:09,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:53:09,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:53:09,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:53:09,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:53:09,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:53:09,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:53:09,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:53:09,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:09,835 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:53:09,838 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:09,838 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:11,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:11,970 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:53:11,974 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:11,974 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:13,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:14,138 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:53:14,144 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:14,144 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:16,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:16,318 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:53:16,322 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:16,322 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:18,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:18,492 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:53:18,496 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:18,496 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:20,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:20,640 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:20,640 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:22,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:22,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:22,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:24,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:24,964 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:24,964 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:26,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:27,107 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:27,108 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:29,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:29,250 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:29,250 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:31,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:31,403 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:31,403 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:33,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:33,552 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:33,552 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:35,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:35,709 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:35,709 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:37,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:37,931 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:37,931 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:39,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:40,085 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:40,086 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:42,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:42,246 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:42,246 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:44,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:44,414 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:44,414 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:46,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:46,575 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:46,575 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:48,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:48,739 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:48,739 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:50,739 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:50,898 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:50,898 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:52,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:53,063 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:53,065 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:55,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:55,213 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:55,213 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:57,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:57,371 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:57,371 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:53:59,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:53:59,529 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:53:59,529 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:01,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:01,687 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:01,687 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:03,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:03,848 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:03,849 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:05,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:05,999 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:06,000 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:08,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:08,158 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:08,158 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:10,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:10,315 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:10,315 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:12,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:12,469 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:12,470 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:14,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:14,625 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:14,625 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:16,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:16,779 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:16,779 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:18,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:18,957 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:18,957 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:20,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:21,138 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:21,138 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:23,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:23,319 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:23,319 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:25,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:25,496 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:25,496 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:27,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:27,667 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:27,668 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:29,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:29,839 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:54:29,839 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:54:31,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:32,289 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:54:32,289 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.2290000916. 2021-12-16 04:54:32,290 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:54:32,292 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:54:32,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:54:32,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:54:32,788 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:54:32,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:54:32,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:54:32,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:54:32,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:54:32,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:54:32,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:54:32,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:54:32,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:33,157 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:54:33,157 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.867000102997. 2021-12-16 04:54:33,158 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:54:33,161 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:54:33,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:54:33,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:54:33,670 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:54:33,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:54:33,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:54:33,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:54:33,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:54:33,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:54:33,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:54:33,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:54:33,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:34,028 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:54:34,028 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.870000123978. 2021-12-16 04:54:34,029 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:54:34,030 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:54:34,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:54:34,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:54:34,364 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:54:34,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:54:34,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:54:34,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:54:34,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:54:34,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:54:34,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:54:34,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:54:34,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:54:34,625 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:54:34,627 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.59700012207. ############################################################ # # RAM FootPrint: 4127.8359375 # ############################################################ 2021-12-16 04:54:34,658 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:54:34,660 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 87 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-16 04:56:01,664 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:56:01,664 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:56:06,667 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:56:07,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:07,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:56:07,710 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:07,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:56:07,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:56:07,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:56:07,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:56:08,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:56:08,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:56:08,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:56:08,451 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:56:08,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:56:08,532 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['207807 memcached\n'], []) 2021-12-16 04:56:08,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:56:08,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:56:08,713 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['207807 memcached\n'], []) 2021-12-16 04:56:08,713 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:56:08,717 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:56:08,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:09,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:56:09,187 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:09,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:56:09,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:56:09,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:56:09,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:56:09,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:56:09,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:56:09,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:56:09,328 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:56:09,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:56:09,400 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['170756 memcached\n'], []) 2021-12-16 04:56:09,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:56:09,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:56:09,546 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['170756 memcached\n'], []) 2021-12-16 04:56:09,546 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:56:09,551 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:56:09,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:09,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:56:10,033 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:10,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:56:10,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:56:10,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:56:10,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:56:10,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:56:10,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:56:10,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:56:10,177 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:56:10,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:56:10,256 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['44435 memcached\n'], []) 2021-12-16 04:56:10,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:56:10,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:56:10,401 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['44435 memcached\n'], []) 2021-12-16 04:56:10,403 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:56:10,404 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:56:10,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:10,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:56:10,887 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:10,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:56:10,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:56:10,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:56:10,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:56:10,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:56:11,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:56:11,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:56:11,058 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:56:11,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:56:11,141 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['250028 memcached\n'], []) 2021-12-16 04:56:11,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:56:11,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:56:11,289 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['250028 memcached\n'], []) 2021-12-16 04:56:11,290 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:56:11,292 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:56:11,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:12,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:56:12,322 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:12,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:56:12,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:56:12,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:56:12,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:56:12,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:56:12,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:56:12,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:56:12,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:56:12,473 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:56:12,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:56:12,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:12,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:12,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:12,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:12,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:12,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:12,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:12,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:12,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:12,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:12,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:12,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:12,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:56:12,961 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:56:12,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:56:12,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:12,997 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:56:13,013 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:56:13,013 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:56:13,016 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:56:13,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:13,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:56:13,463 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:13,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:56:13,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:56:13,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:56:13,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:56:13,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:56:13,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:56:13,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:56:13,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:56:13,625 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:56:13,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:56:13,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:13,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:13,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:13,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:13,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:13,874 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:13,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:13,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:13,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:14,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:14,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:14,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:14,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:14,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:14,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:14,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:14,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:14,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:14,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:14,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:14,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:56:14,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:56:14,604 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:56:14,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:56:14,716 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:56:14,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:56:14,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:56:14,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:56:14,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,900 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:14,926 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:56:14,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:56:14,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:14,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:56:15,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:56:15,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,086 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:56:15,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:56:15,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:56:15,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:56:15,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:56:15,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:56:15,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:56:15,443 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:56:15,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:15,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:56:15,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:56:15,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:56:15,569 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:56:15,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:56:15,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:15,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:56:15,621 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:56:15,621 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:56:15,624 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:56:15,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:16,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:56:16,148 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:16,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:56:16,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:56:16,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:56:16,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:56:16,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:56:16,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:56:16,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:56:16,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:56:16,292 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:56:16,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:56:16,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:16,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:16,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:16,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:16,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:16,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:16,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:16,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:16,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:16,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:16,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:16,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:16,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:16,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:16,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:16,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:16,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:17,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:17,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,181 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:17,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:17,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:17,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:17,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:17,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:17,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:56:17,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:56:17,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:56:17,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:56:17,821 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:17,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:17,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:56:17,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:56:17,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:56:18,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:56:18,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:56:18,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:56:18,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:56:18,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:56:18,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:56:18,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:56:18,505 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:56:18,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:56:18,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:56:18,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:56:18,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:56:18,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:56:18,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:18,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:18,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:56:18,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:56:19,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:56:19,023 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:56:19,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:56:19,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:19,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:56:19,075 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:56:19,075 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:56:19,078 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:56:19,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:19,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:56:19,588 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:19,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:56:19,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:56:19,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:56:19,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:56:19,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:56:19,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:56:19,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:56:19,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:56:19,750 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:56:19,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:56:19,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:19,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:19,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:19,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:56:19,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:19,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:19,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:20,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:56:20,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:20,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:56:20,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:20,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:56:20,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:20,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:56:20,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:20,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:56:20,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:20,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:56:20,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:56:20,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:20,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:56:20,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:56:20,721 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:56:20,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:56:20,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:56:20,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:56:20,769 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:56:20,769 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:56:20,770 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:56:20,773 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:56:20,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:56:21,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:56:21,299 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:56:21,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:56:21,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:56:21,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:56:21,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:56:21,390 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:56:21,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:56:21,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:56:21,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:21,634 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:21,638 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:21,638 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:23,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:23,806 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:23,811 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:23,812 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:25,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:25,976 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:25,982 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:25,982 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:27,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:28,155 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:28,161 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:28,161 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:30,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:30,326 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:30,331 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:30,331 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:32,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:32,513 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:32,517 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:32,517 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:34,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:34,690 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:56:34,694 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:34,694 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:36,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:36,846 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:36,848 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:38,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:38,997 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:38,999 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:40,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:41,140 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:41,141 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:43,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:43,282 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:43,282 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:45,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:45,427 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:45,427 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:47,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:47,572 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:47,572 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:49,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:49,717 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:49,717 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:51,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:51,855 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:51,855 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:53,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:54,006 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:54,007 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:56,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:56,157 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:56,157 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:56:58,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:56:58,312 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:56:58,312 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:00,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:00,469 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:00,469 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:02,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:02,625 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:02,625 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:04,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:04,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:04,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:06,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:06,943 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:06,943 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:08,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:09,096 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:09,096 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:11,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:11,246 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:11,246 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:13,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:13,404 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:13,405 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:15,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:15,569 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:15,569 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:17,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:17,730 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:17,732 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:19,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:19,890 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:19,890 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:21,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:22,049 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:22,049 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:24,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:24,194 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:24,194 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:26,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:26,355 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:26,355 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:28,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:28,513 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:28,513 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:30,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:30,678 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:30,678 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:32,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:32,943 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:32,944 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:34,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:35,104 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:35,105 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:37,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:37,276 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:37,276 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:39,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:39,456 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:39,457 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:41,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:41,624 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:41,624 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:43,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:43,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:43,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:45,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:45,977 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:45,977 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:47,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:48,137 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:57:48,137 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:57:50,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:50,894 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:57:50,894 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.1240000725. 2021-12-16 04:57:50,895 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:57:50,897 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:57:51,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:57:51,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:57:51,423 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:57:51,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:57:51,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:57:51,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:57:51,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:57:51,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:57:51,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:57:51,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:57:51,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:51,798 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:57:51,798 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.902000188828. 2021-12-16 04:57:51,798 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:57:51,801 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:57:51,923 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:57:52,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:57:52,322 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:57:52,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:57:52,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:57:52,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:57:52,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:57:52,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:57:52,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:57:52,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:57:52,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:52,665 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:57:52,665 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.867999792099. 2021-12-16 04:57:52,667 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:57:52,668 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:57:52,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:57:52,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:57:53,013 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:57:53,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:57:53,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:57:53,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:57:53,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:57:53,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:57:53,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:57:53,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:57:53,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:57:53,279 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:57:53,279 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.611999988556. ############################################################ # # RAM FootPrint: 4082.88671875 # ############################################################ 2021-12-16 04:57:53,309 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 04:57:53,311 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 64 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-16 04:58:57,315 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 04:58:57,315 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 04:59:02,318 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:59:02,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:02,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:59:03,033 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:03,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:59:03,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:59:03,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:59:03,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:59:03,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:59:03,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:59:03,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:59:03,216 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:59:03,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:59:03,276 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['209581 memcached\n'], []) 2021-12-16 04:59:03,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 04:59:03,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 04:59:03,437 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['209581 memcached\n'], []) 2021-12-16 04:59:03,437 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:59:03,440 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:59:03,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:03,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:59:03,924 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:03,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:59:03,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:59:03,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:59:03,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:59:04,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:59:04,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:59:04,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:59:04,062 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:59:04,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:59:04,117 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['171713 memcached\n'], []) 2021-12-16 04:59:04,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 04:59:04,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 04:59:04,236 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['171713 memcached\n'], []) 2021-12-16 04:59:04,236 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:59:04,240 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:59:04,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:04,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:59:04,757 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:04,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:59:04,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:59:04,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:59:04,821 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:59:04,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:59:04,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:59:04,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:59:04,908 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:59:04,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:59:04,983 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['45244 memcached\n'], []) 2021-12-16 04:59:04,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 04:59:05,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 04:59:05,082 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['45244 memcached\n'], []) 2021-12-16 04:59:05,082 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:59:05,085 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:59:05,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:05,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:59:05,558 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:05,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:59:05,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:59:05,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:59:05,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:59:05,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:59:05,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:59:05,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:59:05,736 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 04:59:05,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:59:05,822 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['251544 memcached\n'], []) 2021-12-16 04:59:05,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 04:59:05,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 04:59:05,963 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['251544 memcached\n'], []) 2021-12-16 04:59:05,963 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:59:05,966 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 04:59:06,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:07,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 04:59:07,357 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:07,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 04:59:07,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 04:59:07,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 04:59:07,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 04:59:07,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 04:59:07,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 04:59:07,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 04:59:07,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:59:07,506 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 04:59:07,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:59:07,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:07,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:07,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:07,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:07,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:07,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:07,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:07,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:07,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:07,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:07,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:07,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:07,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:07,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:07,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:08,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:08,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:59:08,038 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 04:59:08,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:59:08,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:08,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:59:08,084 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 04:59:08,084 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 04:59:08,085 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 04:59:08,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:08,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 04:59:08,549 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:08,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 04:59:08,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 04:59:08,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 04:59:08,617 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 04:59:08,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 04:59:08,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 04:59:08,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 04:59:08,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:59:08,717 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 04:59:08,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:59:08,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:08,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:08,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:08,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:08,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:08,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:08,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:09,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:09,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,104 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:09,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:09,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:09,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:09,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:09,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:09,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:09,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:09,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:09,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:09,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:09,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:09,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:59:09,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:59:09,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:59:09,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:59:09,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,803 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:59:09,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:59:09,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:59:09,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:59:09,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:09,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:09,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:59:09,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:59:09,987 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:59:10,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:59:10,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:59:10,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:59:10,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:59:10,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:59:10,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:59:10,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:59:10,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:59:10,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:59:10,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:59:10,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:59:10,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:10,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 04:59:10,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 04:59:10,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:59:10,529 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 04:59:10,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:59:10,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:10,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:59:10,562 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 04:59:10,562 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 04:59:10,565 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:59:10,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:11,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:59:11,088 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:11,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:59:11,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:59:11,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:59:11,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:59:11,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:59:11,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:59:11,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:59:11,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:59:11,220 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 04:59:11,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:59:11,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:11,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:11,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:11,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:11,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:11,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:11,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:11,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:11,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:11,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:11,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:11,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:11,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:11,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:11,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:11,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:11,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:11,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:11,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:59:12,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 04:59:12,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:59:12,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 04:59:12,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:59:12,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 04:59:12,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:59:12,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 04:59:12,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:59:12,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 04:59:12,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:59:12,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 04:59:12,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:59:12,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 04:59:12,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:59:12,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 04:59:12,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:59:12,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 04:59:12,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:59:12,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 04:59:12,658 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:12,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:59:12,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 04:59:12,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:59:12,720 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 04:59:12,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:59:12,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:12,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:59:12,750 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 04:59:12,752 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 04:59:12,755 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 04:59:12,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:13,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 04:59:13,244 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:13,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 04:59:13,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 04:59:13,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 04:59:13,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 04:59:13,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 04:59:13,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 04:59:13,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 04:59:13,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 04:59:13,414 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 04:59:13,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 04:59:13,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:13,502 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:13,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:13,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 04:59:13,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:13,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:13,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:13,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 04:59:13,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:13,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:13,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:13,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 04:59:13,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:13,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:13,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:13,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 04:59:14,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:14,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:14,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:14,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 04:59:14,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:14,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:14,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:14,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 04:59:14,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:14,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:14,390 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:14,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 04:59:14,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:14,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 04:59:14,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:14,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 04:59:14,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 04:59:14,582 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 04:59:14,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 04:59:14,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 04:59:14,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 04:59:14,628 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 04:59:14,628 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 04:59:14,630 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 04:59:14,634 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 04:59:14,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 04:59:15,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 04:59:15,118 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 04:59:15,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 04:59:15,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 04:59:15,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 04:59:15,165 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 04:59:15,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 04:59:15,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 04:59:15,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 04:59:15,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:15,378 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:59:15,382 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:15,382 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:17,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:17,516 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:59:17,520 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:17,522 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:19,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:19,621 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:59:19,627 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:19,627 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:21,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:21,802 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:59:21,809 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:21,809 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:23,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:23,986 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 04:59:23,990 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:23,990 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:25,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:26,141 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:26,141 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:28,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:28,299 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:28,301 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:30,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:30,451 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:30,451 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:32,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:32,621 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:32,621 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:34,622 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:34,786 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:34,786 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:36,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:36,931 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:36,931 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:38,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:39,072 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:39,072 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:41,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:41,216 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:41,216 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:43,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:43,371 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:43,371 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:45,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:45,523 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:45,523 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:47,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:47,690 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:47,690 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:49,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:49,838 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:49,838 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:51,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:51,997 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:51,999 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:53,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:54,157 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:54,158 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:56,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:56,318 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:56,319 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 04:59:58,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 04:59:58,470 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 04:59:58,470 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:00,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:00,622 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:00,624 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:02,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:02,783 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:02,785 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:04,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:04,951 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:04,953 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:06,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:07,111 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:07,111 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:09,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:09,256 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:09,257 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:11,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:11,417 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:11,417 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:13,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:13,576 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:13,578 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:15,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:15,733 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:15,734 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:17,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:17,884 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:17,885 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:19,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:20,040 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:20,040 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:22,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:22,194 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:22,194 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:24,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:24,352 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:24,352 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:26,354 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:27,012 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:27,012 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:29,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:29,186 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:29,186 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:31,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:31,359 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:31,361 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:33,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:33,532 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:33,532 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:35,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:35,707 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:35,709 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:00:37,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:37,913 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:00:37,914 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.2829999924. 2021-12-16 05:00:37,914 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:00:37,915 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:00:38,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:00:38,318 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:00:38,404 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:00:38,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:00:38,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:00:38,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:00:38,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:00:38,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:00:38,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:00:38,535 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:00:38,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:38,769 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:00:38,769 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.854000091553. 2021-12-16 05:00:38,769 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:00:38,772 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:00:38,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:00:39,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:00:39,301 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:00:39,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:00:39,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:00:39,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:00:39,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:00:39,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:00:39,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:00:39,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:00:39,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:39,650 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:00:39,650 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.881000041962. 2021-12-16 05:00:39,651 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:00:39,653 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:00:39,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:00:39,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:00:39,974 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:00:39,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:00:39,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:00:40,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:00:40,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:00:40,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:00:40,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:00:40,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:00:40,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:40,243 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:40,243 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:00:42,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:42,398 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:00:42,398 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:00:44,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:00:44,776 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:00:44,776 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.125. ############################################################ # # RAM FootPrint: 4063.22265625 # ############################################################ 2021-12-16 05:00:44,815 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:00:44,815 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 61 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-16 05:01:45,819 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:01:45,819 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:01:50,822 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:01:51,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:51,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:01:51,976 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:51,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:01:51,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:01:52,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:01:52,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:01:52,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:01:52,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:01:52,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:01:52,138 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:01:52,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:01:52,198 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['211515 memcached\n'], []) 2021-12-16 05:01:52,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:01:52,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:01:52,349 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['211515 memcached\n'], []) 2021-12-16 05:01:52,349 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:01:52,352 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:01:52,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:52,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:01:52,859 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:52,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:01:52,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:01:52,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:01:52,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:01:52,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:01:52,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:01:52,987 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:01:53,012 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:01:53,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:01:53,098 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['172661 memcached\n'], []) 2021-12-16 05:01:53,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:01:53,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:01:53,234 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['172661 memcached\n'], []) 2021-12-16 05:01:53,234 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:01:53,239 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:01:53,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:53,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:01:53,770 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:53,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:01:53,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:01:53,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:01:53,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:01:53,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:01:53,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:01:53,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:01:53,930 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:01:53,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:01:54,007 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['46072 memcached\n'], []) 2021-12-16 05:01:54,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:01:54,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:01:54,109 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['46072 memcached\n'], []) 2021-12-16 05:01:54,111 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:01:54,112 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:01:54,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:54,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:01:54,569 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:54,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:01:54,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:01:54,622 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:01:54,650 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:01:54,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:01:54,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:01:54,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:01:54,732 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:01:54,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:01:54,793 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['253083 memcached\n'], []) 2021-12-16 05:01:54,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:01:54,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:01:54,944 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['253083 memcached\n'], []) 2021-12-16 05:01:54,946 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:01:54,947 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:01:55,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:55,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:01:55,441 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:55,443 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:01:55,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:01:55,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:01:55,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:01:55,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:01:55,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:01:55,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:01:55,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:01:55,553 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:01:55,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:01:55,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:55,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:55,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:55,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:55,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:55,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:55,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:55,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:55,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:55,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:55,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:55,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:55,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:55,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:55,954 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:01:55,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:01:55,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:01:55,993 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:01:55,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:01:56,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:56,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:01:56,036 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:01:56,036 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:01:56,039 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:01:56,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:56,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:01:56,493 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:56,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:01:56,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:01:56,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:01:56,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:01:56,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:01:56,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:01:56,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:01:56,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:01:56,657 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:01:56,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:01:56,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:56,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:56,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:56,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:56,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:56,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:56,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:56,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:56,959 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:56,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:57,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:57,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:01:57,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:01:57,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,181 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:01:57,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:01:57,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:01:57,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:01:57,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:01:57,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:01:57,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:01:57,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:01:57,566 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:01:57,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:01:57,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:01:57,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:01:57,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:01:57,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:01:57,859 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:01:57,926 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:01:57,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:57,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:57,997 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:01:58,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:01:58,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:01:58,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:01:58,145 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:01:58,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:01:58,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:01:58,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:01:58,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:01:58,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:01:58,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:01:58,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:01:58,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,553 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:01:58,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:01:58,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:58,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:01:58,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:01:58,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:01:58,701 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:01:58,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:01:58,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:58,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:01:58,752 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:01:58,752 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:01:58,755 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:01:58,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:01:59,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:01:59,265 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:01:59,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:01:59,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:01:59,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:01:59,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:01:59,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:01:59,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:01:59,369 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:01:59,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:01:59,401 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:01:59,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:01:59,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:59,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:59,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:59,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:01:59,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:59,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:59,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:59,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:01:59,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:59,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:59,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:59,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:01:59,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:01:59,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:01:59,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:01:59,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:02:00,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:02:00,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:02:00,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:02:00,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:02:00,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:02:00,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:02:00,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:02:00,341 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:02:00,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:02:00,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:02:00,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:02:00,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:02:00,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:02:00,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:02:00,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:02:00,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:02:00,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,900 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:00,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:02:00,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:02:00,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:00,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:02:01,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:02:01,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:02:01,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:02:01,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:02:01,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:02:01,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:02:01,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:02:01,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,515 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:02:01,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:02:01,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:01,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:02:01,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:02:01,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:02:01,700 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:02:01,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:02:01,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:01,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:02:01,740 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:02:01,742 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:02:01,744 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:02:01,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:02:02,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:02:02,200 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:02:02,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:02:02,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:02:02,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:02:02,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:02:02,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:02:02,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:02:02,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:02:02,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:02:02,346 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:02:02,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:02:02,390 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:02,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:02,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:02:02,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:02:02,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:02,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:02,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:02:02,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:02:02,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:02,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:02,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:02:02,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:02:02,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:02,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:02,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:02:02,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:02:02,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:02,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:02,980 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:02:03,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:02:03,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:03,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:03,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:02:03,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:02:03,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:03,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:03,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:02:03,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:02:03,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:03,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:02:03,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:02:03,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:02:03,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:02:03,433 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:02:03,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:02:03,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:02:03,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:02:03,476 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:02:03,476 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:02:03,477 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:02:03,480 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:02:03,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:02:03,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:02:03,973 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:02:03,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:02:03,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:02:04,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:02:04,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:02:04,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:02:04,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:02:04,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:02:04,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:04,344 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:04,348 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:04,348 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:06,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:06,526 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:06,529 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:06,530 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:08,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:08,701 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:08,707 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:08,707 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:10,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:10,875 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:10,881 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:10,881 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:12,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:13,056 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:13,062 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:13,062 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:15,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:15,232 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:02:15,236 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:15,236 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:17,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:17,451 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:17,451 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:19,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:19,609 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:19,609 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:21,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:21,756 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:21,756 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:23,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:23,903 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:23,903 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:25,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:26,062 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:26,062 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:28,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:28,223 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:28,224 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:30,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:30,388 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:30,388 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:32,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:32,529 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:32,529 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:34,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:34,736 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:34,736 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:36,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:36,898 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:36,898 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:38,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:39,049 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:39,049 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:41,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:41,200 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:41,201 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:43,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:43,365 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:43,365 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:45,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:45,523 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:45,523 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:47,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:47,694 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:47,694 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:49,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:49,852 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:49,854 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:51,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:52,005 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:52,006 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:54,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:54,170 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:54,170 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:56,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:56,328 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:56,328 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:02:58,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:02:58,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:02:58,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:00,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:00,648 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:00,648 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:02,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:02,796 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:02,796 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:04,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:04,961 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:04,961 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:06,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:07,115 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:07,115 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:09,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:09,269 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:09,270 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:11,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:11,431 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:11,431 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:13,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:13,592 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:13,592 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:15,594 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:15,821 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:15,822 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:17,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:18,052 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:18,052 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:20,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:20,210 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:20,210 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:22,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:22,372 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:22,374 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:24,374 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:24,562 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:24,563 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:26,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:26,723 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:26,723 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:28,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:28,905 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:03:28,907 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:03:30,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:31,109 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:03:31,111 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.6330001354. 2021-12-16 05:03:31,111 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:03:31,114 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:03:31,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:03:31,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:03:31,621 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:03:31,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:03:31,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:03:31,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:03:31,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:03:31,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:03:31,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:03:31,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:03:31,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:31,963 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:03:31,963 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.851999998093. 2021-12-16 05:03:31,964 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:03:31,967 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:03:32,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:03:32,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:03:32,457 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:03:32,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:03:32,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:03:32,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:03:32,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:03:32,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:03:32,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:03:32,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:03:32,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:32,802 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:03:32,803 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.838999986649. 2021-12-16 05:03:32,805 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:03:32,806 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:03:32,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:03:33,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:03:33,683 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:03:33,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:03:33,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:03:33,790 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:03:33,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:03:33,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:03:33,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:03:33,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:03:34,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:03:34,237 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:03:34,239 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.43400001526. ############################################################ # # RAM FootPrint: 4046.02734375 # ############################################################ 2021-12-16 05:03:34,278 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:03:34,279 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 115 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-16 05:05:29,282 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:05:29,282 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:05:34,285 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:05:34,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:34,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:05:34,746 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:34,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:05:34,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:05:34,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:05:34,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:05:34,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:05:34,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:05:34,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:05:34,826 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:05:34,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:05:34,872 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['213236 memcached\n'], []) 2021-12-16 05:05:34,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:05:34,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:05:34,976 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['213236 memcached\n'], []) 2021-12-16 05:05:34,976 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:05:34,980 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:05:35,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:35,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:05:35,493 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:35,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:05:35,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:05:35,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:05:35,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:05:35,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:05:35,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:05:35,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:05:35,578 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:05:35,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:05:35,621 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['173608 memcached\n'], []) 2021-12-16 05:05:35,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:05:35,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:05:35,717 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['173608 memcached\n'], []) 2021-12-16 05:05:35,717 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:05:35,720 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:05:35,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:36,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:05:36,200 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:36,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:05:36,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:05:36,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:05:36,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:05:36,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:05:36,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:05:36,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:05:36,315 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:05:36,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:05:36,358 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['46906 memcached\n'], []) 2021-12-16 05:05:36,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:05:36,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:05:36,503 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['46906 memcached\n'], []) 2021-12-16 05:05:36,503 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:05:36,506 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:05:36,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:36,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:05:37,015 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:37,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:05:37,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:05:37,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:05:37,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:05:37,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:05:37,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:05:37,142 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:05:37,161 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:05:37,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:05:37,210 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['254672 memcached\n'], []) 2021-12-16 05:05:37,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:05:37,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:05:37,358 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['254672 memcached\n'], []) 2021-12-16 05:05:37,358 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:05:37,359 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:05:37,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:39,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:05:39,312 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:39,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:05:39,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:05:39,354 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:05:39,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:05:39,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:05:39,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:05:39,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:05:39,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:05:39,476 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:05:39,502 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:05:39,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:39,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:39,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:39,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:39,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:39,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:39,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:39,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:39,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:39,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:39,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:39,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:39,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:39,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:39,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:39,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:39,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:05:39,927 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:05:39,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:05:39,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:39,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:05:39,964 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:05:39,964 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:05:39,967 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:05:40,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:40,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:05:40,382 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:40,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:05:40,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:05:40,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:05:40,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:05:40,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:05:40,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:05:40,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:05:40,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:05:40,503 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:05:40,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:05:40,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:40,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:40,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:40,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:40,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:40,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:40,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:40,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:40,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:40,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:40,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:40,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:40,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:40,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:40,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:40,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:40,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:41,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:41,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:41,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:41,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:41,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:41,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:41,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:41,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:41,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:41,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:05:41,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:05:41,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:05:41,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:05:41,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:05:41,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:05:41,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:05:41,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:05:41,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:05:41,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:05:41,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:41,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:41,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:05:42,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:05:42,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:42,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:05:42,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:05:42,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:42,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:05:42,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:05:42,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:42,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:05:42,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:05:42,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:42,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:05:42,566 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:05:42,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:42,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:05:42,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:05:42,706 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:05:42,730 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:05:42,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:05:42,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:42,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:05:42,803 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:05:42,803 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:05:42,806 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:05:42,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:43,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:05:43,328 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:43,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:05:43,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:05:43,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:05:43,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:05:43,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:05:43,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:05:43,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:05:43,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:05:43,470 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:05:43,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:05:43,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:43,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:43,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:43,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:43,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:43,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:43,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:43,836 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:43,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:43,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:43,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:43,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:44,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:44,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:44,173 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:44,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:44,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:44,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:44,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:44,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:44,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:44,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:44,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:44,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:44,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:44,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:44,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:05:45,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:05:45,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:05:45,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:05:45,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:05:45,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:05:45,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,413 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:05:45,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:05:45,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:05:45,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:05:45,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:05:45,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:05:45,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:45,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:45,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:05:45,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:05:46,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:46,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:46,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:05:46,122 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:05:46,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:46,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:46,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:05:46,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:05:46,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:46,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:46,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:05:46,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:05:46,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:05:46,426 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:05:46,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:05:46,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:46,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:05:46,476 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:05:46,476 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:05:46,479 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:05:46,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:46,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:05:46,990 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:46,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:05:47,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:05:47,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:05:47,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:05:47,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:05:47,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:05:47,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:05:47,145 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:05:47,170 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:05:47,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:05:47,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:47,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:05:47,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:47,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:05:47,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:47,617 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:05:47,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,678 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:47,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:05:47,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:47,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:05:47,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:47,936 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:47,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:48,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:05:48,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:48,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:48,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:48,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:05:48,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:48,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:48,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:48,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:05:48,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:48,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:05:48,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:48,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:05:48,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:05:48,441 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:05:48,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:05:48,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:05:48,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:05:48,490 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:05:48,490 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:05:48,490 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:05:48,493 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:05:48,632 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:05:48,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:05:49,040 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:05:49,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:05:49,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:05:49,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:05:49,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:05:49,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:05:49,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:05:49,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:05:49,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:05:49,398 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:05:49,405 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:05:49,405 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:05:51,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:05:51,569 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:05:51,573 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:05:51,573 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:05:53,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:05:53,756 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:05:53,760 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:05:53,760 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:05:55,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:05:55,936 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:05:55,940 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:05:55,940 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:05:57,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:05:58,118 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:05:58,124 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:05:58,124 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:00,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:00,305 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:00,309 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:00,309 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:02,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:02,476 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:02,479 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:02,480 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:04,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:04,664 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:04,667 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:04,667 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:06,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:06,845 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:06,849 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:06,851 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:08,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:09,009 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:09,013 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:09,015 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:11,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:11,178 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:06:11,184 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:11,184 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:13,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:13,332 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:13,332 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:15,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:15,482 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:15,482 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:17,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:17,638 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:17,638 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:19,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:19,780 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:19,780 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:21,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:21,924 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:21,924 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:23,926 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:24,072 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:24,073 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:26,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:26,217 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:26,219 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:28,219 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:28,361 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:28,361 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:30,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:30,520 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:30,522 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:32,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:32,674 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:32,674 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:34,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:34,839 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:34,839 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:36,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:37,000 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:37,000 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:39,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:39,160 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:39,160 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:41,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:41,321 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:41,321 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:43,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:43,474 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:43,476 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:45,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:45,637 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:45,638 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:47,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:47,789 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:47,789 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:49,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:49,944 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:49,946 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:51,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:52,102 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:52,102 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:54,104 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:54,259 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:54,260 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:56,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:56,411 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:56,411 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:06:58,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:06:58,569 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:06:58,569 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:00,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:00,732 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:00,733 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:02,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:02,901 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:02,901 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:04,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:05,075 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:05,075 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:07,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:07,266 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:07,266 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:09,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:09,427 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:09,427 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:11,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:11,609 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:11,611 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:13,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:13,786 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:13,786 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:15,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:15,960 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:15,960 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:17,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:18,127 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:18,127 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:20,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:20,315 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:20,315 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:22,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:22,487 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:07:22,489 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:07:24,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:24,921 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:07:24,923 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.4320001602. 2021-12-16 05:07:24,924 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:07:24,924 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:07:25,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:07:25,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:07:25,427 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:07:25,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:07:25,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:07:25,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:07:25,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:07:25,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:07:25,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:07:25,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:07:25,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:25,750 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:07:25,750 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.827999830246. 2021-12-16 05:07:25,753 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:07:25,753 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:07:25,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:07:26,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:07:26,242 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:07:26,242 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:07:26,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:07:26,278 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:07:26,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:07:26,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:07:26,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:07:26,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:07:26,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:26,553 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:07:26,555 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.802000045776. 2021-12-16 05:07:26,555 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:07:26,556 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:07:26,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:07:26,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:07:27,013 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:07:27,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:07:27,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:07:27,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:07:27,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:07:27,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:07:27,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:07:27,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:07:27,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:07:27,530 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:07:27,530 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.975999832153. ############################################################ # # RAM FootPrint: 4648.1328125 # ############################################################ 2021-12-16 05:07:27,565 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:07:27,565 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 78 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-16 05:08:45,569 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:08:45,569 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:08:50,572 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:08:50,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:51,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:08:51,732 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:51,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:08:51,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:08:51,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:08:51,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:08:51,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:08:51,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:08:51,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:08:51,861 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:08:51,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:08:51,917 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['214676 memcached\n'], []) 2021-12-16 05:08:51,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:08:52,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:08:52,061 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['214676 memcached\n'], []) 2021-12-16 05:08:52,061 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:08:52,063 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:08:52,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:52,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:08:52,573 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:52,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:08:52,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:08:52,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:08:52,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:08:52,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:08:52,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:08:52,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:08:52,720 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:08:52,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:08:52,786 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['174622 memcached\n'], []) 2021-12-16 05:08:52,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:08:52,836 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:08:52,907 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['174622 memcached\n'], []) 2021-12-16 05:08:52,907 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:08:52,911 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:08:53,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:53,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:08:53,433 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:53,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:08:53,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:08:53,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:08:53,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:08:53,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:08:53,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:08:53,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:08:53,585 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:08:53,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:08:53,661 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['47801 memcached\n'], []) 2021-12-16 05:08:53,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:08:53,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:08:53,785 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['47801 memcached\n'], []) 2021-12-16 05:08:53,785 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:08:53,788 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:08:53,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:54,157 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:08:54,259 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:54,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:08:54,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:08:54,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:08:54,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:08:54,367 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:08:54,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:08:54,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:08:54,436 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:08:54,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:08:54,523 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['256272 memcached\n'], []) 2021-12-16 05:08:54,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:08:54,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:08:54,667 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['256272 memcached\n'], []) 2021-12-16 05:08:54,667 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:08:54,670 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:08:54,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:56,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:08:56,224 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:56,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:08:56,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:08:56,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:08:56,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:08:56,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:08:56,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:08:56,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:08:56,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:08:56,359 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:08:56,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:08:56,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:56,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:56,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:08:56,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:08:56,553 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:56,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:56,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:08:56,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:08:56,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:56,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:56,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:08:56,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:08:56,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:56,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:56,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:08:56,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:08:56,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:08:56,926 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:08:56,926 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:08:56,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:56,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:08:56,964 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:08:56,966 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:08:56,967 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:08:57,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:08:57,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:08:57,415 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:08:57,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:08:57,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:08:57,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:08:57,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:08:57,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:08:57,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:08:57,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:08:57,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:08:57,559 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:08:57,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:08:57,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:57,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:57,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:08:57,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:08:57,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:57,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:57,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:08:57,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:08:57,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,007 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:08:58,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:08:58,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:08:58,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:08:58,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:08:58,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:08:58,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:08:58,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:08:58,566 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,602 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:08:58,668 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:08:58,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:08:58,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:08:58,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:58,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:08:58,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:08:58,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:58,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:08:59,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:08:59,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:08:59,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:08:59,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:08:59,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:08:59,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:08:59,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:08:59,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:08:59,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:08:59,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:08:59,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:08:59,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:08:59,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:08:59,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:08:59,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:08:59,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:08:59,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:08:59,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:08:59,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:08:59,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:08:59,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:08:59,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:00,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:00,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:09:00,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:09:00,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:09:00,063 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:09:00,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:09:00,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:00,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:09:00,101 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:09:00,101 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:09:00,104 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:09:00,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:09:00,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:09:00,624 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:09:00,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:09:00,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:09:00,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:09:00,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:09:00,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:09:00,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:09:00,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:09:00,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:09:00,767 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:09:00,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:09:00,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:00,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:00,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:09:00,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:09:00,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:09:01,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:09:01,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:09:01,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:09:01,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:09:01,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:09:01,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:09:01,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:09:01,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,637 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:09:01,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:09:01,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:09:01,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:09:01,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:01,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:01,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:09:01,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:09:02,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:09:02,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:09:02,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:09:02,283 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:09:02,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:09:02,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:09:02,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:09:02,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:09:02,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:09:02,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:09:02,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:09:02,747 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:09:02,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:02,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:09:02,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:09:02,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:02,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:03,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:09:03,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:09:03,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:03,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:03,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:09:03,187 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:09:03,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:03,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:03,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:09:03,354 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:09:03,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:03,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:03,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:09:03,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:09:03,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:09:03,530 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:09:03,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:09:03,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:03,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:09:03,584 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:09:03,585 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:09:03,588 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:09:03,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:09:03,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:09:04,053 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:09:04,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:09:04,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:09:04,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:09:04,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:09:04,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:09:04,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:09:04,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:09:04,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:09:04,219 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:09:04,249 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:09:04,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:04,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:09:04,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:09:04,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:04,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:09:04,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:09:04,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:04,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:09:04,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:09:04,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:04,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:09:04,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:09:04,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:04,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:09:04,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:09:04,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:04,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:05,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:09:05,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:09:05,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:05,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:05,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:09:05,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:09:05,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:05,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:05,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:09:05,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:09:05,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:05,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:09:05,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:09:05,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:09:05,456 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:09:05,473 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:09:05,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:09:05,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:09:05,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:09:05,525 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:09:05,525 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:09:05,525 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:09:05,528 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:09:05,645 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:09:05,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:09:06,036 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:09:06,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:09:06,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:09:06,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:09:06,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:09:06,124 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:09:06,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:09:06,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:09:06,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:06,335 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:06,338 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:06,339 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:08,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:08,516 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:08,520 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:08,520 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:10,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:10,676 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:10,680 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:10,680 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:12,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:12,819 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:12,825 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:12,825 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:14,826 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:15,000 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:15,005 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:15,006 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:17,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:17,187 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:09:17,191 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:17,191 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:19,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:19,335 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:19,336 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:21,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:21,496 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:21,496 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:23,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:23,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:23,641 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:25,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:25,789 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:25,789 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:27,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:27,927 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:27,927 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:29,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:30,082 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:30,082 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:32,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:32,229 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:32,229 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:34,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:34,365 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:34,365 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:36,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:36,523 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:36,523 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:38,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:38,680 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:38,681 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:40,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:40,845 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:40,845 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:42,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:43,000 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:43,000 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:45,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:45,164 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:45,164 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:47,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:47,326 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:47,326 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:49,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:49,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:49,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:51,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:51,647 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:51,648 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:53,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:53,802 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:53,802 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:55,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:55,954 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:55,954 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:09:57,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:09:58,119 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:09:58,119 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:00,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:00,286 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:00,286 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:02,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:02,446 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:02,446 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:04,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:04,608 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:04,609 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:06,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:06,763 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:06,763 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:08,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:08,924 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:08,924 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:10,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:11,073 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:11,073 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:13,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:13,221 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:13,223 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:15,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:15,375 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:15,375 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:17,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:17,525 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:17,525 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:19,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:20,269 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:20,270 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:22,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:22,536 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:22,536 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:24,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:24,779 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:24,779 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:26,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:27,348 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:27,348 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:29,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:29,536 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:10:29,538 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:10:31,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:32,463 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:10:32,463 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 86.9389998913. 2021-12-16 05:10:32,466 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:10:32,467 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:10:32,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:10:32,881 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:10:32,970 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:10:32,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:10:32,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:10:33,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:10:33,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:10:33,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:10:33,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:10:33,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:10:33,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:33,315 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:10:33,315 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.848999977112. 2021-12-16 05:10:33,315 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:10:33,318 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:10:33,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:10:33,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:10:33,828 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:10:33,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:10:33,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:10:33,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:10:33,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:10:33,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:10:33,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:10:33,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:10:33,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:34,174 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:10:34,174 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.859999895096. 2021-12-16 05:10:34,176 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:10:34,177 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:10:34,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:10:34,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:10:35,022 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:10:35,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:10:35,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:10:35,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:10:35,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:10:35,163 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:10:35,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:10:35,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:10:35,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:10:35,474 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:10:35,474 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.29999995232. ############################################################ # # RAM FootPrint: 4616.71875 # ############################################################ 2021-12-16 05:10:35,509 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:10:35,509 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 84 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-16 05:11:59,513 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:11:59,513 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:12:04,516 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:12:04,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:05,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:12:05,551 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:05,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:12:05,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:12:05,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:12:05,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:12:05,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:12:05,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:12:05,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:12:05,723 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:12:05,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:12:05,799 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['216506 memcached\n'], []) 2021-12-16 05:12:05,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:12:05,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:12:05,921 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['216506 memcached\n'], []) 2021-12-16 05:12:05,921 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:12:05,926 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:12:06,058 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:06,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:12:06,438 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:06,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:12:06,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:12:06,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:12:06,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:12:06,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:12:06,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:12:06,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:12:06,576 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:12:06,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:12:06,641 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['175601 memcached\n'], []) 2021-12-16 05:12:06,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:12:06,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:12:06,753 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['175601 memcached\n'], []) 2021-12-16 05:12:06,753 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:12:06,756 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:12:06,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:07,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:12:07,305 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:07,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:12:07,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:12:07,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:12:07,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:12:07,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:12:07,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:12:07,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:12:07,454 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:12:07,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:12:07,545 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['48633 memcached\n'], []) 2021-12-16 05:12:07,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:12:07,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:12:07,680 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['48633 memcached\n'], []) 2021-12-16 05:12:07,681 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:12:07,684 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:12:07,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:08,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:12:08,140 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:08,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:12:08,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:12:08,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:12:08,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:12:08,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:12:08,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:12:08,278 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:12:08,299 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:12:08,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:12:08,382 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['257836 memcached\n'], []) 2021-12-16 05:12:08,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:12:08,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:12:08,510 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['257836 memcached\n'], []) 2021-12-16 05:12:08,512 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:12:08,513 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:12:08,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:11,946 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:12:12,030 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:12,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:12:12,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:12:12,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:12:12,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:12:12,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:12:12,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:12:12,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:12:12,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:12:12,191 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:12:12,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:12:12,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:12,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:12,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:12,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:12,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:12,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:12,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:12,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:12,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:12,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:12,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:12,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:12,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:12,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:12,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:12,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:12,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:12:12,779 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:12:12,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:12:12,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:12,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:12:12,832 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:12:12,832 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:12:12,835 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:12:12,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:13,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:12:13,298 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:13,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:12:13,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:12:13,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:12:13,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:12:13,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:12:13,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:12:13,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:12:13,461 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:12:13,482 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:12:13,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:12:13,529 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:13,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:13,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:13,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:13,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:13,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:13,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:13,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:13,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:13,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:13,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:13,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:13,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:14,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:14,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,165 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:14,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:14,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,265 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:14,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:14,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,362 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:14,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:14,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:14,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:14,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,522 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:14,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:14,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,622 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:12:14,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:12:14,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:12:14,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:12:14,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:12:14,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:12:14,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:14,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:12:14,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:12:14,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:14,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:12:15,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:12:15,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:12:15,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:12:15,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:12:15,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:12:15,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,293 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:12:15,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:12:15,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:12:15,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:12:15,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:12:15,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:12:15,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:15,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:12:15,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:12:15,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:12:15,562 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:12:15,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:12:15,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:15,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:12:15,607 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:12:15,607 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:12:15,609 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:12:15,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:15,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:12:16,035 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:16,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:12:16,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:12:16,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:12:16,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:12:16,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:12:16,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:12:16,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:12:16,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:12:16,217 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:12:16,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:12:16,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:16,303 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:16,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:16,390 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:16,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:16,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:16,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:16,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:16,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:16,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:16,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:16,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:16,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:16,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:16,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:16,842 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:16,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:16,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:16,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:16,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:16,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,048 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:17,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:17,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:17,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:17,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:17,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:17,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:17,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:17,464 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,497 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:12:17,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:12:17,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,632 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:12:17,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:12:17,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:12:17,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:12:17,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:17,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:12:17,905 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:12:17,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:17,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:12:18,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:12:18,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:12:18,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:12:18,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:12:18,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:12:18,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:12:18,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:12:18,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:12:18,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:12:18,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:12:18,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:12:18,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:18,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:12:18,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:12:18,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:12:18,882 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:12:18,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:12:18,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:18,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:12:18,937 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:12:18,937 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:12:18,940 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:12:19,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:19,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:12:19,448 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:19,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:12:19,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:12:19,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:12:19,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:12:19,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:12:19,555 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:12:19,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:12:19,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:12:19,598 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:12:19,622 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:12:19,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:19,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:19,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:19,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:12:19,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:19,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:19,752 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:19,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:12:19,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:19,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:19,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:19,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:12:19,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:19,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:19,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:19,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:12:19,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:19,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:19,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:19,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:12:20,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:20,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:20,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:20,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:12:20,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:20,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:20,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:20,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:12:20,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:20,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:20,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:20,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:12:20,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:20,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:12:20,276 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:20,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:12:20,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:12:20,316 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:12:20,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:12:20,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:12:20,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:12:20,357 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:12:20,357 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:12:20,357 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:12:20,359 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:12:20,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:12:20,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:12:20,867 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:12:20,867 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:12:20,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:12:20,907 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:12:20,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:12:20,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:12:20,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:12:20,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:12:21,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:21,171 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:12:21,176 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:21,176 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:23,176 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:23,354 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:12:23,358 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:23,358 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:25,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:25,539 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:12:25,543 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:25,543 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:27,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:27,720 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:12:27,723 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:27,723 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:29,724 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:29,898 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:12:29,903 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:29,903 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:31,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:32,063 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:32,065 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:34,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:34,242 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:34,243 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:36,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:36,405 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:36,405 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:38,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:38,561 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:38,561 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:40,561 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:40,707 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:40,707 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:42,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:42,854 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:42,855 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:44,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:44,997 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:44,999 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:46,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:47,190 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:47,190 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:49,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:49,345 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:49,346 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:51,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:51,505 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:51,505 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:53,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:53,664 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:53,664 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:55,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:55,823 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:55,825 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:57,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:12:57,980 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:12:57,982 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:12:59,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:00,148 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:00,148 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:02,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:02,305 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:02,305 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:04,305 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:04,460 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:04,460 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:06,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:06,614 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:06,615 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:08,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:08,766 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:08,766 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:10,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:10,917 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:10,917 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:12,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:13,079 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:13,079 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:15,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:15,242 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:15,243 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:17,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:17,394 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:17,394 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:19,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:19,552 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:19,552 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:21,553 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:22,092 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:22,092 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:24,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:24,296 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:24,296 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:26,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:26,470 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:26,470 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:28,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:28,642 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:28,642 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:30,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:30,822 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:30,823 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:32,823 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:33,076 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:33,076 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:35,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:35,579 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:35,579 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:37,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:37,744 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:37,746 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:39,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:40,253 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:40,253 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:42,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:42,423 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:13:42,423 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:13:44,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:44,861 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:13:44,862 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 84.5059998035. 2021-12-16 05:13:44,864 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:13:44,865 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:13:44,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:13:45,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:13:45,336 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:13:45,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:13:45,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:13:45,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:13:45,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:13:45,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:13:45,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:13:45,469 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:13:45,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:45,697 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:13:45,697 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832999944687. 2021-12-16 05:13:45,697 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:13:45,700 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:13:45,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:13:46,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:13:46,200 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:13:46,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:13:46,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:13:46,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:13:46,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:13:46,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:13:46,313 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:13:46,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:13:46,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:46,569 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:13:46,571 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.873000144958. 2021-12-16 05:13:46,572 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:13:46,572 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:13:46,739 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:13:47,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:13:47,188 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:13:47,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:13:47,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:13:47,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:13:47,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:13:47,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:13:47,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:13:47,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:13:47,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:13:47,611 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:13:47,612 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.04199981689. ############################################################ # # RAM FootPrint: 4565.36328125 # ############################################################ 2021-12-16 05:13:47,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:13:47,642 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-16 05:14:53,645 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:14:53,645 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:14:58,648 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:14:59,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:14:59,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:14:59,974 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:14:59,974 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:14:59,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:15:00,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:15:00,033 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:15:00,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:15:00,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:15:00,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:15:00,102 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:15:00,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:15:00,174 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['217921 memcached\n'], []) 2021-12-16 05:15:00,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:15:00,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:15:00,423 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['217921 memcached\n'], []) 2021-12-16 05:15:00,423 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:15:00,424 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:15:00,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:00,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:15:00,898 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:00,900 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:15:00,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:15:00,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:15:00,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:15:00,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:15:01,009 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:15:01,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:15:01,046 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:15:01,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:15:01,105 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['176597 memcached\n'], []) 2021-12-16 05:15:01,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:15:01,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:15:01,223 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['176597 memcached\n'], []) 2021-12-16 05:15:01,223 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:15:01,226 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:15:01,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:01,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:15:01,733 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:01,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:15:01,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:15:01,776 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:15:01,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:15:01,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:15:01,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:15:01,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:15:01,871 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:15:01,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:15:01,931 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['49496 memcached\n'], []) 2021-12-16 05:15:01,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:15:02,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:15:02,078 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['49496 memcached\n'], []) 2021-12-16 05:15:02,078 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:15:02,082 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:15:02,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:02,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:15:02,552 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:02,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:15:02,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:15:02,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:15:02,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:15:02,651 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:15:02,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:15:02,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:15:02,713 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:15:02,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:15:02,796 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['259397 memcached\n'], []) 2021-12-16 05:15:02,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:15:02,859 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:15:02,930 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['259397 memcached\n'], []) 2021-12-16 05:15:02,930 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:15:02,933 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:15:03,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:04,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:15:04,259 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:04,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:15:04,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:15:04,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:15:04,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:15:04,344 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:15:04,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:15:04,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:15:04,395 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:15:04,413 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:15:04,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:15:04,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:04,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:04,545 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:04,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:04,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:04,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:04,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:04,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:04,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:04,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:04,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:04,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:04,838 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:04,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:04,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:04,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:04,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:15:04,927 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:15:04,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:15:04,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:04,956 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:15:04,970 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:15:04,970 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:15:04,973 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:15:05,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:05,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:15:05,364 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:05,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:15:05,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:15:05,405 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:15:05,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:15:05,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:15:05,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:15:05,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:15:05,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:15:05,539 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:15:05,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:15:05,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:05,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:05,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:05,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:05,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:05,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:05,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:05,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:05,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:05,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:06,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:06,079 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:06,181 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:06,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:06,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:06,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:06,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:06,424 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:06,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:06,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,536 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,558 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:06,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:06,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:06,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:06,670 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,711 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:15:06,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:15:06,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:15:06,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:15:06,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:06,862 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:06,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:15:06,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:15:06,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:15:07,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:15:07,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:15:07,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:15:07,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:15:07,354 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:15:07,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:15:07,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:15:07,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:15:07,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:15:07,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:15:07,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:15:07,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,877 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:07,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:15:07,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:15:07,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:07,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:08,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:15:08,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:15:08,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:15:08,082 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:15:08,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:15:08,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:08,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:15:08,137 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:15:08,137 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:15:08,138 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:15:08,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:08,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:15:08,661 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:08,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:15:08,681 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:15:08,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:15:08,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:15:08,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:15:08,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:15:08,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:15:08,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:15:08,802 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:15:08,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:15:08,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:08,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:08,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:09,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:09,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,094 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:09,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:09,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,302 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:09,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:09,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,408 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:09,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:09,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:09,618 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:09,650 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:09,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:09,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:09,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:09,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:09,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:09,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:10,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:10,084 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:10,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:10,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:15:10,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:15:10,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:15:10,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:15:10,512 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:15:10,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:15:10,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,684 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:15:10,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:15:10,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:10,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:15:10,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:15:10,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:10,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:15:11,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:15:11,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:15:11,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:15:11,249 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:15:11,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:15:11,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:15:11,513 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:15:11,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:15:11,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:15:11,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:11,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:15:11,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:15:11,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:15:11,785 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:15:11,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:15:11,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:11,816 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:15:11,832 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:15:11,832 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:15:11,835 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:15:11,934 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:12,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:15:12,299 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:12,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:15:12,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:15:12,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:15:12,364 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:15:12,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:15:12,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:15:12,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:15:12,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:15:12,473 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:15:12,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:15:12,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:12,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:12,582 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:12,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:15:12,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:12,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:12,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:12,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:15:12,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:12,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:12,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:12,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:15:12,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:12,943 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:12,984 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:13,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:15:13,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:13,105 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:13,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:15:13,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:13,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:13,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:15:13,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:13,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:13,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:15:13,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:13,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:13,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:15:13,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:15:13,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:13,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:15:13,671 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:15:13,690 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:15:13,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:15:13,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:15:13,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:15:13,744 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:15:13,744 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:15:13,744 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:15:13,747 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:15:13,878 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:15:14,160 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:15:14,242 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:15:14,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:15:14,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:15:14,286 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:15:14,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:15:14,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:15:14,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:15:14,365 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:15:14,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:14,480 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:14,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:14,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:16,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:16,657 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:16,661 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:16,661 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:18,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:18,835 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:18,841 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:18,841 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:20,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:21,012 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:21,016 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:21,016 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:23,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:23,190 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:23,194 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:23,194 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:25,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:25,375 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:15:25,380 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:25,380 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:27,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:27,529 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:27,530 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:29,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:29,683 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:29,683 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:31,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:31,836 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:31,836 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:33,836 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:33,986 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:33,986 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:35,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:36,128 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:36,130 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:38,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:38,278 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:38,278 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:40,278 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:40,421 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:40,421 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:42,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:42,576 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:42,576 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:44,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:44,729 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:44,730 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:46,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:46,884 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:46,884 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:48,884 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:49,038 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:49,038 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:51,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:51,191 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:51,193 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:53,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:53,345 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:53,346 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:55,346 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:55,503 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:55,503 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:57,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:57,657 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:57,657 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:15:59,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:15:59,815 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:15:59,815 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:01,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:01,966 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:01,966 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:03,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:04,121 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:04,122 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:06,122 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:06,279 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:06,279 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:08,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:08,430 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:08,430 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:10,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:10,586 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:10,588 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:12,588 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:12,753 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:12,753 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:14,755 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:14,910 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:14,910 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:16,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:17,073 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:17,073 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:19,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:19,234 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:19,234 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:21,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:21,387 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:21,387 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:23,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:23,546 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:23,546 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:25,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:25,703 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:25,703 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:27,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:27,852 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:27,854 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:29,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:30,013 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:30,013 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:32,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:32,851 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:32,851 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:34,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:35,023 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:35,023 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:37,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:37,190 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:16:37,190 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:16:39,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:39,822 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:16:39,822 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 86.0780000687. 2021-12-16 05:16:39,823 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:16:39,825 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:16:39,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:16:40,227 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:16:40,315 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:16:40,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:16:40,338 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:16:40,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:16:40,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:16:40,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:16:40,423 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:16:40,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:16:40,463 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:40,694 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:16:40,694 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871000051498. 2021-12-16 05:16:40,696 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:16:40,697 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:16:40,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:16:41,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:16:41,194 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:16:41,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:16:41,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:16:41,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:16:41,253 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:16:41,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:16:41,288 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:16:41,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:16:41,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:41,461 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:16:41,461 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.766999959946. 2021-12-16 05:16:41,463 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:16:41,464 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:16:41,624 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:16:41,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:16:41,940 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:16:41,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:16:42,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:16:42,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:16:42,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:16:42,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:16:42,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:16:42,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:16:42,256 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:16:42,467 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:16:42,467 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.0039999485. ############################################################ # # RAM FootPrint: 4524.2734375 # ############################################################ 2021-12-16 05:16:42,500 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:16:42,500 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 102 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-16 05:18:24,503 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:18:24,503 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:18:29,505 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:18:29,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:30,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:18:30,111 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:30,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:18:30,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:18:30,145 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:18:30,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:18:30,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:18:30,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:18:30,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:18:30,223 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:18:30,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:18:30,275 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['220123 memcached\n'], []) 2021-12-16 05:18:30,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:18:30,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:18:30,359 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['220123 memcached\n'], []) 2021-12-16 05:18:30,359 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:18:30,362 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:18:30,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:30,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:18:30,862 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:30,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:18:30,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:18:30,901 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:18:30,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:18:30,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:18:30,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:18:30,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:18:30,989 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:18:30,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:18:31,039 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['177562 memcached\n'], []) 2021-12-16 05:18:31,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:18:31,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:18:31,164 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['177562 memcached\n'], []) 2021-12-16 05:18:31,164 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:18:31,167 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:18:31,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:31,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:18:31,678 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:31,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:18:31,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:18:31,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:18:31,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:18:31,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:18:31,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:18:31,798 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:18:31,819 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:18:31,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:18:31,882 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['50318 memcached\n'], []) 2021-12-16 05:18:31,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:18:31,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:18:32,033 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['50318 memcached\n'], []) 2021-12-16 05:18:32,033 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:18:32,036 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:18:32,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:32,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:18:32,470 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:32,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:18:32,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:18:32,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:18:32,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:18:32,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:18:32,592 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:18:32,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:18:32,638 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:18:32,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:18:32,723 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['260937 memcached\n'], []) 2021-12-16 05:18:32,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:18:32,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:18:32,862 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['260937 memcached\n'], []) 2021-12-16 05:18:32,862 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:18:32,867 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:18:33,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:35,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:18:35,997 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:35,997 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:18:36,013 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:18:36,028 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:18:36,046 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:18:36,063 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:18:36,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:18:36,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:18:36,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:18:36,124 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:18:36,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:18:36,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:36,221 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:36,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:36,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,285 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:36,308 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:36,325 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:36,349 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:36,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:36,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:36,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:36,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:36,525 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:36,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:36,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:36,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:36,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:18:36,611 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:18:36,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:18:36,627 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:36,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:18:36,651 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:18:36,651 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:18:36,654 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:18:36,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:37,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:18:37,092 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:37,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:18:37,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:18:37,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:18:37,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:18:37,165 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:18:37,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:18:37,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:18:37,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:18:37,226 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:18:37,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:18:37,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:37,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:37,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:37,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:37,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:37,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:37,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:37,635 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:37,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:37,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,747 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:37,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:37,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:37,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:37,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:37,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:37,966 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:37,994 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:38,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:38,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,096 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:38,165 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:38,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:38,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:38,323 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:38,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:38,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:18:38,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:18:38,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,619 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:18:38,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:18:38,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:18:38,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:18:38,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:18:38,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:18:38,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:38,880 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:38,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:18:38,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:18:38,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:18:39,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:18:39,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:18:39,260 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:18:39,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:18:39,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:18:39,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,535 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:18:39,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:18:39,602 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:18:39,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:18:39,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,737 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:39,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:18:39,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:18:39,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:18:39,845 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:18:39,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:18:39,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:39,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:18:39,905 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:18:39,907 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:18:39,910 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:18:40,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:40,328 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:18:40,407 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:40,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:18:40,426 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:18:40,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:18:40,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:18:40,479 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:18:40,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:18:40,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:18:40,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:18:40,542 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:18:40,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:18:40,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:40,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:40,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:40,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:40,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:40,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:40,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:40,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:40,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:40,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,006 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:41,038 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:41,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,104 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,144 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:41,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:41,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:41,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:41,322 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:41,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:41,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,460 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:41,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:41,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,585 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:41,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:41,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:41,821 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:41,854 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:41,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:41,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:18:41,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:18:41,999 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,036 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:18:42,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:18:42,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,209 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:18:42,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:18:42,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,306 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,348 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:18:42,377 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:18:42,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,434 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:18:42,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:18:42,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:18:42,660 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:18:42,693 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,785 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:18:42,819 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:18:42,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:42,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:42,923 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:18:42,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:18:42,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:43,019 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:43,056 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:18:43,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:18:43,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:43,155 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:43,196 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:18:43,223 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:18:43,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:43,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:43,309 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:18:43,334 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:18:43,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:18:43,380 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:18:43,380 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:18:43,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:43,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:18:43,441 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:18:43,441 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:18:43,444 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:18:43,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:43,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:18:43,943 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:43,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:18:43,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:18:43,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:18:44,005 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:18:44,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:18:44,043 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:18:44,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:18:44,082 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:18:44,104 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:18:44,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:18:44,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,279 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:44,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:18:44,342 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,410 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:44,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:18:44,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,530 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:44,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:18:44,568 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:44,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:18:44,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:44,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:18:44,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,766 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:44,782 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:18:44,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,821 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:44,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:18:44,872 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:44,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:18:44,948 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:44,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:18:44,983 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:45,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:18:45,015 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:18:45,026 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:18:45,028 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:18:45,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:18:45,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:18:45,065 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:18:45,065 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:18:45,065 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:18:45,068 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:18:45,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:18:45,500 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:18:45,586 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:18:45,586 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:18:45,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:18:45,621 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:18:45,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:18:45,663 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:18:45,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:18:45,691 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:18:45,709 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:45,917 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:18:45,921 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:45,921 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:47,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:48,088 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:18:48,091 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:48,092 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:50,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:50,239 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:50,240 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:52,240 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:52,385 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:52,385 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:54,387 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:54,525 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:54,525 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:56,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:56,676 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:56,676 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:18:58,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:18:58,821 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:18:58,821 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:00,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:00,970 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:00,970 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:02,970 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:03,118 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:03,119 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:05,119 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:05,259 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:05,259 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:07,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:07,411 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:07,411 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:09,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:09,563 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:09,565 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:11,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:11,726 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:11,726 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:13,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:13,881 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:13,882 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:15,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:16,039 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:16,039 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:18,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:18,200 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:18,201 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:20,201 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:20,358 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:20,359 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:22,359 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:22,517 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:22,517 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:24,517 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:24,678 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:24,680 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:26,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:26,832 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:26,832 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:28,832 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:28,987 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:28,989 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:30,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:31,131 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:31,131 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:33,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:33,292 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:33,292 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:35,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:35,450 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:35,450 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:37,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:37,611 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:37,611 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:39,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:39,775 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:39,775 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:41,775 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:41,938 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:41,938 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:43,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:44,190 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:44,190 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:46,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:46,369 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:46,369 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:48,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:48,540 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:48,540 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:50,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:50,720 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:50,720 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:52,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:52,892 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:52,892 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:54,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:55,068 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:55,069 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:57,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:57,230 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:57,230 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:19:59,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:19:59,765 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:19:59,765 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:20:01,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:02,022 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:20:02,022 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 76.9570000172. 2021-12-16 05:20:02,023 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:20:02,025 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:20:02,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:20:02,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:20:02,522 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:20:02,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:20:02,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:20:02,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:20:02,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:20:02,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:20:02,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:20:02,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:20:02,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:02,836 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:20:02,836 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.814999818802. 2021-12-16 05:20:02,838 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:20:02,839 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:20:02,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:20:03,273 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:20:03,367 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:20:03,368 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:20:03,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:20:03,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:20:03,430 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:20:03,454 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:20:03,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:20:03,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:20:03,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:03,690 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:20:03,690 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.852999925613. 2021-12-16 05:20:03,690 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:20:03,693 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:20:03,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:20:03,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:20:04,026 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:20:04,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:20:04,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:20:04,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:20:04,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:20:04,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:20:04,099 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:20:04,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:20:04,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:04,288 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:20:04,289 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:20:06,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:06,437 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:20:06,437 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:20:08,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:08,598 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:20:08,599 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:20:10,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:10,756 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:20:10,756 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:20:12,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:20:13,055 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:20:13,056 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 9.36500000954. ############################################################ # # RAM FootPrint: 4454.13671875 # ############################################################ 2021-12-16 05:20:13,088 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:20:13,088 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-16 05:21:59,092 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:21:59,092 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:22:04,094 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:22:04,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:04,438 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:22:04,484 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:04,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:22:04,496 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:22:04,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:22:04,519 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:22:04,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:22:04,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:22:04,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:22:04,562 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:22:04,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:22:04,615 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['221456 memcached\n'], []) 2021-12-16 05:22:04,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:22:04,657 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:22:04,733 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['221456 memcached\n'], []) 2021-12-16 05:22:04,733 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:22:04,736 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:22:04,852 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:05,085 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:22:05,141 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:05,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:22:05,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:22:05,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:22:05,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:22:05,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:22:05,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:22:05,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:22:05,237 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:22:05,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:22:05,299 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['178583 memcached\n'], []) 2021-12-16 05:22:05,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:22:05,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:22:05,418 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['178583 memcached\n'], []) 2021-12-16 05:22:05,418 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:22:05,423 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:22:05,538 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:05,844 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:22:05,933 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:05,933 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:22:05,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:22:05,971 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:22:05,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:22:06,016 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:22:06,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:22:06,053 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:22:06,078 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:22:06,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:22:06,168 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['51221 memcached\n'], []) 2021-12-16 05:22:06,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:22:06,244 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:22:06,316 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['51221 memcached\n'], []) 2021-12-16 05:22:06,316 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:22:06,319 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:22:06,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:06,696 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:22:06,792 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:06,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:22:06,818 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:22:06,839 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:22:06,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:22:06,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:22:06,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:22:06,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:22:06,951 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:22:06,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:22:07,032 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['262516 memcached\n'], []) 2021-12-16 05:22:07,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:22:07,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:22:07,184 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['262516 memcached\n'], []) 2021-12-16 05:22:07,186 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:22:07,190 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:22:07,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:08,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:22:08,388 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:08,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:22:08,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:22:08,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:22:08,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:22:08,459 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:22:08,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:22:08,487 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:22:08,505 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:22:08,519 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:22:08,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:22:08,565 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:08,607 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:08,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:08,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:08,717 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:08,756 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:08,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:08,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:08,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:08,885 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:08,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:08,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:08,989 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:09,028 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:09,068 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:09,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:09,125 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:09,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:09,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:09,188 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:09,206 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:22:09,220 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:22:09,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:22:09,234 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:09,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:22:09,263 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:22:09,263 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:22:09,266 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:22:09,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:09,641 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:22:09,727 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:09,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:22:09,749 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:22:09,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:22:09,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:22:09,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:22:09,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:22:09,845 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:22:09,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:22:09,878 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:22:09,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:22:09,927 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:09,977 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,035 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:10,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:10,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,147 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:10,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:10,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,312 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,352 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:10,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:10,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,493 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:10,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:10,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,576 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,604 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:10,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:10,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:10,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:10,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:10,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:10,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:10,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:10,941 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:10,964 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:10,996 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,039 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:22:11,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:22:11,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:22:11,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:22:11,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,292 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:22:11,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:22:11,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:22:11,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:22:11,466 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:22:11,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:22:11,573 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,601 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,628 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:22:11,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:22:11,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:22:11,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:22:11,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:22:11,874 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:22:11,895 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:11,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:11,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:22:11,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:22:12,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:12,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:12,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:22:12,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:22:12,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:12,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:12,130 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:22:12,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:22:12,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:12,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:12,197 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 05:22:12,211 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 05:22:12,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:22:12,237 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:22:12,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:22:12,252 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:12,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:22:12,280 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:22:12,280 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:22:12,283 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:22:12,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:12,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:22:12,786 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:12,788 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:22:12,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:22:12,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:22:12,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:22:12,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:22:12,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:22:12,898 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:22:12,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:22:12,930 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:22:12,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:22:12,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:13,102 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:13,131 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,226 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:13,263 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:13,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,382 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:13,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:13,444 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,523 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:13,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:13,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:13,683 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:13,710 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:13,809 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:13,834 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:13,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:13,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:13,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:13,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,025 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,072 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:14,107 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:14,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,186 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:14,267 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:14,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,381 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:22:14,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:22:14,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,477 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:22:14,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:22:14,548 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:22:14,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:22:14,625 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:22:14,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:22:14,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,713 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,732 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:22:14,746 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:22:14,759 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,780 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,803 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:22:14,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:22:14,841 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,864 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:22:14,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:22:14,920 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:14,940 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:14,961 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:22:14,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:22:14,992 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:15,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:15,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:22:15,073 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:22:15,089 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:15,111 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:15,132 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:22:15,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:22:15,167 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:15,184 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:15,200 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:22:15,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:22:15,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:22:15,246 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:22:15,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:22:15,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:15,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:22:15,285 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:22:15,285 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:22:15,288 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:22:15,413 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:15,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:22:15,786 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:15,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:22:15,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:22:15,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:22:15,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:22:15,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:22:15,892 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:22:15,910 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:22:15,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:22:15,950 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:22:15,979 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:22:16,000 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,065 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:16,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:22:16,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,173 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:16,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:22:16,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,247 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:16,299 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:22:16,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:16,415 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:22:16,436 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,471 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:16,533 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:22:16,559 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,599 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:16,674 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:22:16,698 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:16,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:22:16,858 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:16,917 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:16,944 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:22:16,967 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:16,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:22:17,023 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:17,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:22:17,069 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:22:17,088 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:22:17,088 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:22:17,108 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:22:17,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:22:17,144 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:22:17,144 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:22:17,144 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:22:17,147 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:22:17,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:22:17,571 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:22:17,654 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:22:17,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:22:17,676 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:22:17,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:22:17,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:22:17,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:22:17,762 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:22:17,779 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:22:17,801 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:17,967 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:17,973 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:17,973 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:19,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:20,151 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:20,154 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:20,154 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:22,154 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:22,328 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:22,332 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:22,332 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:24,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:24,512 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:24,516 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:24,516 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:26,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:26,686 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:26,690 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:26,690 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:28,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:28,857 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:28,861 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:28,861 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:30,861 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:31,045 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:31,049 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:31,049 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:33,049 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:33,226 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:33,232 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:33,232 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:35,233 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:35,401 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:35,407 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:35,407 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:37,407 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:37,579 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:37,584 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:37,584 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:39,584 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:39,749 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:39,753 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:39,753 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:41,753 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:41,927 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:41,930 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:41,930 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:43,931 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:44,108 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:22:44,112 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:44,114 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:46,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:46,256 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:46,257 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:48,257 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:48,404 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:48,404 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:50,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:50,556 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:50,556 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:52,556 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:52,704 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:52,704 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:54,704 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:54,855 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:54,855 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:56,855 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:57,010 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:57,012 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:22:59,012 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:22:59,170 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:22:59,170 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:01,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:01,332 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:01,332 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:03,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:03,476 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:03,476 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:05,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:05,637 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:05,637 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:07,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:07,795 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:07,795 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:09,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:09,956 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:09,957 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:11,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:12,111 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:12,112 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:14,112 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:14,272 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:14,272 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:16,272 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:16,431 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:16,431 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:18,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:18,591 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:18,591 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:20,591 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:20,743 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:20,744 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:22,744 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:22,895 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:22,897 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:24,897 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:25,075 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:25,076 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:27,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:27,223 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:27,224 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:29,224 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:29,384 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:29,384 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:31,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:31,539 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:31,539 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:33,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:33,698 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:33,698 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:35,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:35,855 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:35,857 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:37,857 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:38,019 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:38,020 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:40,020 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:40,181 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:40,183 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:42,183 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:42,395 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:42,395 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:44,397 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:44,552 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:44,552 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:46,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:46,798 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:46,799 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:48,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:48,954 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:48,954 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:50,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:51,127 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:51,127 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:53,127 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:53,296 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:53,296 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:55,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:55,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:55,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:57,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:57,654 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:23:57,654 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:23:59,654 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:23:59,786 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:23:59,786 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 102.64199996. 2021-12-16 05:23:59,786 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:23:59,789 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:23:59,908 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:24:00,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:24:00,295 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:24:00,295 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:24:00,316 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:24:00,335 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:24:00,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:24:00,384 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:24:00,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:24:00,421 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:24:00,441 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:24:00,618 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:24:00,618 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.831000089645. 2021-12-16 05:24:00,618 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:24:00,621 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:24:00,736 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:24:01,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:24:01,118 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:24:01,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:24:01,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:24:01,153 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:24:01,171 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:24:01,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:24:01,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:24:01,217 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:24:01,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:24:01,378 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:24:01,378 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.759999990463. 2021-12-16 05:24:01,380 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:24:01,381 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:24:01,447 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:24:01,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:24:01,707 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:24:01,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:24:01,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:24:01,734 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:24:01,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:24:01,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:24:01,783 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:24:01,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:24:01,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:24:01,976 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:24:01,976 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:24:03,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:24:04,138 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:24:04,138 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 05:24:06,138 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:24:06,539 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:24:06,540 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.16199994087. ############################################################ # # RAM FootPrint: 4515.9453125 # ############################################################ 2021-12-16 05:24:06,569 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 05:24:06,569 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-16 05:25:53,572 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 05:25:53,572 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 05:25:58,575 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:25:58,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:25:58,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:25:59,010 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:25:59,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:25:59,032 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:25:59,055 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:25:59,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:25:59,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:25:59,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:25:59,137 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:25:59,151 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:25:59,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:25:59,214 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['223879 memcached\n'], []) 2021-12-16 05:25:59,214 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 05:25:59,269 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 05:25:59,335 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['223879 memcached\n'], []) 2021-12-16 05:25:59,335 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:25:59,338 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:25:59,418 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:25:59,655 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:25:59,719 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:25:59,719 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:25:59,733 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:25:59,744 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:25:59,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:25:59,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:25:59,803 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:25:59,813 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:25:59,825 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.68 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:25:59,825 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:25:59,871 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.68 - Process info before sending signal: (['179609 memcached\n'], []) 2021-12-16 05:25:59,871 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: kill -9 $(pgrep memcached) 2021-12-16 05:25:59,913 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: pgrep -l memcached 2021-12-16 05:25:59,970 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.68 - Process info after sending signal: (['179609 memcached\n'], []) 2021-12-16 05:25:59,970 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:25:59,973 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:26:00,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:00,391 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:26:00,473 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:00,473 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:26:00,492 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:26:00,507 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:26:00,528 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:26:00,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:26:00,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:26:00,578 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:26:00,596 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:26:00,596 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:26:00,686 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['52142 memcached\n'], []) 2021-12-16 05:26:00,686 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 05:26:00,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 05:26:00,831 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['52142 memcached\n'], []) 2021-12-16 05:26:00,831 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:26:00,832 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:26:00,960 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:01,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:26:01,309 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:01,311 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:26:01,336 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:26:01,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:26:01,385 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:26:01,411 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:26:01,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:26:01,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:26:01,476 | infra | DEBUG | Thread-3 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 05:26:01,476 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:26:01,549 | infra | DEBUG | Thread-3 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['264176 memcached\n'], []) 2021-12-16 05:26:01,549 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 05:26:01,605 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 05:26:01,678 | infra | DEBUG | Thread-3 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['264176 memcached\n'], []) 2021-12-16 05:26:01,678 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:26:01,681 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:26:01,793 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:03,232 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:26:03,319 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:03,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:26:03,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:26:03,358 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:26:03,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:26:03,398 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:26:03,413 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:26:03,427 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:26:03,446 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:26:03,460 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 05:26:03,486 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:26:03,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:03,551 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:03,598 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:03,632 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:03,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:03,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:03,743 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:03,773 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:03,799 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:03,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:03,875 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:03,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:03,928 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:03,963 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:04,002 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:04,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:04,051 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:04,076 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:04,101 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:04,118 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:04,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:26:04,150 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 05:26:04,151 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:26:04,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:04,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:26:04,194 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 05:26:04,194 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:26:04,197 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:26:04,301 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:04,543 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:26:04,612 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:04,614 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:26:04,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:26:04,644 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:26:04,665 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:26:04,687 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:26:04,701 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:26:04,714 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:26:04,729 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:26:04,743 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 05:26:04,767 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:26:04,786 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:04,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:04,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:04,880 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:04,904 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:04,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:04,969 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:04,997 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:05,022 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,052 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:05,114 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:05,134 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,170 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,207 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:05,243 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:05,275 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,315 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,357 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:05,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:05,417 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,453 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,494 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:05,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:05,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,575 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:05,638 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:05,664 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,720 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:05,744 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:05,769 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:26:05,849 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:26:05,869 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:05,894 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:05,930 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:26:05,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:26:05,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,045 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:26:06,071 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:26:06,091 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,115 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,150 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:26:06,178 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:26:06,204 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,230 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,270 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:26:06,296 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:26:06,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:26:06,394 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:26:06,414 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,440 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:26:06,490 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:26:06,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,535 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:26:06,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:26:06,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,647 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,673 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:26:06,694 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:26:06,716 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,742 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,772 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:26:06,796 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:26:06,822 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,887 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:26:06,911 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:26:06,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:06,957 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:06,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 05:26:06,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 05:26:07,010 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:26:07,026 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 05:26:07,026 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:26:07,042 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:07,059 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:26:07,075 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 05:26:07,075 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:26:07,078 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:26:07,193 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:07,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:26:07,595 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:07,595 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:26:07,612 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:26:07,630 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:26:07,648 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:26:07,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:26:07,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:26:07,707 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:26:07,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:26:07,746 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 05:26:07,770 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:26:07,792 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:07,836 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:07,882 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:07,918 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:07,947 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:07,990 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:08,078 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:08,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,148 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,190 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:08,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:08,246 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,282 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,321 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:08,351 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:08,375 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,413 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:08,480 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:08,506 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,542 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,581 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:08,609 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:08,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,697 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:08,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:08,750 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,778 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,806 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:08,831 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:08,851 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:08,890 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:08,938 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:08,973 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:09,003 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,040 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,081 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:26:09,109 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 05:26:09,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,164 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:26:09,229 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 05:26:09,259 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,298 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,339 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:26:09,371 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 05:26:09,401 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,437 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,474 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:26:09,499 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 05:26:09,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,589 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:26:09,615 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 05:26:09,640 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,680 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,727 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:26:09,763 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 05:26:09,815 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:09,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:09,915 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:26:09,950 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 05:26:09,982 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:10,017 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:10,061 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:26:10,092 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 05:26:10,122 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:10,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:10,194 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:26:10,220 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 05:26:10,250 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:10,290 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:10,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:26:10,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 05:26:10,388 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:10,420 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:10,457 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:26:10,484 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 05:26:10,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:26:10,532 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 05:26:10,532 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:26:10,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:10,572 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:26:10,588 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 05:26:10,588 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:26:10,591 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:26:10,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:10,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:26:11,073 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:11,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:26:11,098 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:26:11,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:26:11,140 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:26:11,161 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:26:11,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:26:11,191 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:26:11,213 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 05:26:11,236 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 05:26:11,266 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 05:26:11,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,331 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:11,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:11,403 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 05:26:11,428 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:11,510 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:11,540 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 05:26:11,569 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,602 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:11,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:11,661 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 05:26:11,688 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:11,760 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:11,789 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 05:26:11,812 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,848 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:11,891 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:11,924 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 05:26:11,951 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:11,993 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:12,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:12,062 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 05:26:12,095 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:12,135 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:12,177 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:12,210 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 05:26:12,239 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:12,280 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:12,326 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:12,361 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 05:26:12,392 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:12,431 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 05:26:12,470 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:12,503 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 05:26:12,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 05:26:12,546 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 05:26:12,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 05:26:12,563 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 05:26:12,579 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 05:26:12,595 | test | DEBUG | Thread-3 | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 05:26:12,595 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:26:12,596 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:26:12,601 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 05:26:12,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:26:13,030 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 05:26:13,121 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:26:13,121 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 05:26:13,141 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 05:26:13,158 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 05:26:13,180 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 05:26:13,203 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 05:26:13,219 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 05:26:13,237 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 05:26:13,262 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:13,374 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:13,378 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:13,378 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:15,378 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:15,546 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:15,552 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:15,552 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:17,552 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:17,686 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:17,690 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:17,690 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:19,690 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:19,796 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:19,801 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:19,802 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:21,802 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:21,980 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:21,984 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:21,984 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:23,986 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:24,163 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:24,167 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:24,167 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:26,168 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:26,339 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:26,344 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:26,344 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:28,345 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:28,522 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:28,525 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:28,526 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:30,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:30,694 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:30,698 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:30,698 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:32,700 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:32,861 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:32,867 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:32,867 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:34,868 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:35,019 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:35,028 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:35,029 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:37,029 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:37,194 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:37,197 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:37,197 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:39,198 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:39,365 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:39,368 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:39,368 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:41,369 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:41,539 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:41,545 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:41,545 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:43,546 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:43,719 | test | WARNING | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 05:26:43,723 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:43,723 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:45,723 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:45,865 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:45,865 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:47,867 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:48,065 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:48,066 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:50,066 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:50,214 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:50,214 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:52,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:52,372 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:52,372 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:54,372 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:54,519 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:54,519 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:56,520 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:56,667 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:56,667 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:26:58,667 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:26:58,828 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:26:58,828 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:00,828 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:00,976 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:00,976 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:02,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:03,174 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:03,174 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:05,174 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:05,331 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:05,332 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:07,332 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:07,483 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:07,483 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:09,483 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:09,641 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:09,642 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:11,642 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:11,795 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:11,795 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:13,795 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:13,953 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:13,953 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:15,953 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:16,117 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:16,117 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:18,117 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:18,288 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:18,289 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:20,289 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:20,447 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:20,447 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:22,448 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:22,607 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:22,608 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:24,608 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:24,756 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:24,757 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:26,757 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:26,913 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:26,914 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:28,914 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:29,073 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:29,073 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:31,075 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:31,236 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:31,236 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:33,236 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:33,400 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:33,400 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:35,400 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:35,561 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:35,562 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:37,562 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:37,726 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:37,726 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:39,726 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:40,404 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:40,404 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:42,404 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:42,634 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:42,634 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:44,634 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:44,805 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:44,805 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:46,805 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:46,973 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:46,973 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:48,976 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:49,144 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:49,144 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:51,145 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:51,318 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:51,319 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:53,319 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:53,480 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:53,482 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:55,482 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:55,653 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:55,653 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:57,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:27:57,835 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 05:27:57,835 | test | INFO | Thread-3 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 05:27:59,835 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:28:01,338 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 05:28:01,339 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 108.743000031. 2021-12-16 05:28:01,341 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:28:01,342 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 05:28:01,451 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:28:01,730 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 05:28:01,811 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:28:01,811 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 05:28:01,829 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 05:28:01,846 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 05:28:01,865 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 05:28:01,888 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 05:28:01,903 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 05:28:01,921 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 05:28:01,937 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:28:02,079 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 05:28:02,079 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.740000009537. 2021-12-16 05:28:02,079 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:28:02,085 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 05:28:02,216 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:28:02,516 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 05:28:02,609 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:28:02,611 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 05:28:02,631 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 05:28:02,653 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 05:28:02,677 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 05:28:02,703 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 05:28:02,721 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 05:28:02,740 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 05:28:02,765 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:28:02,986 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 05:28:02,987 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.907999992371. 2021-12-16 05:28:02,987 | test | DEBUG | Thread-3 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 05:28:02,992 | infra | DEBUG | Thread-3 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 05:28:03,128 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 05:28:03,355 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 05:28:03,431 | infra | DEBUG | Thread-3 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 05:28:03,433 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 05:28:03,450 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 05:28:03,467 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 05:28:03,489 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 05:28:03,509 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 05:28:03,526 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 05:28:03,539 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 05:28:03,556 | infra | DEBUG | Thread-3 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 05:28:03,766 | infra | DEBUG | Thread-3 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 05:28:03,766 | test | INFO | Thread-3 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.779000043869. 2021-12-16 05:28:03,770 | test | DEBUG | Thread-3 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-16 05:28:03,770 | test | DEBUG | Thread-3 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 10: Swap with Loading of docs # ############################################################ KV nodes in cluster: ['172.23.110.65', '172.23.110.68', '172.23.110.67'] CBAS nodes in cluster: [] INDEX nodes in cluster: [] FTS nodes in cluster: [] QUERY nodes in cluster: [] EVENTING nodes in cluster: [] AVAILABLE nodes for cluster: ['172.23.110.66'] Servers coming in : ['172.23.110.66'] with services: ['kv'] Servers going out : ['172.23.110.68'] NEW AVAILABLE nodes for cluster: ['172.23.110.68'] 2021-12-16 05:33:03,803 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[172.23.110.66]_OUT=[172.23.110.68]_1639661583.77 2021-12-16 05:33:03,803 | infra | INFO | pool-3-thread-4 | [task:start_task:90] Thread 'Rebalance_task_IN=[172.23.110.66]_OUT=[172.23.110.68]_1639661583.77' started 2021-12-16 05:33:03,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L543 2021-12-16 05:33:03,818 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep for 60s for rebalance to start 2021-12-16 05:33:03,822 | test | DEBUG | pool-3-thread-4 | [rest_client:add_node:1184] Adding remote node 172.23.110.66:8091 to cluster 172.23.110.67:8091 2021-12-16 05:33:07,497 | test | DEBUG | pool-3-thread-4 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-16 05:33:07,499 | test | DEBUG | pool-3-thread-4 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-16 05:33:07,552 | test | DEBUG | pool-3-thread-4 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 05:33:07,555 | test | INFO | pool-3-thread-4 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 16.6956229334 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 13.4167180234 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 19.9494594974 | --- OUT ---> | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 14.4614608584 | Cluster node | | 172.23.110.66 | ['kv'] | | | <--- IN --- | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-16 05:33:07,563 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:07,563 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:07,578 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:07,578 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:07,579 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:33:17,589 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:17,591 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:17,608 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:17,611 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:17,611 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:33:27,621 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:27,622 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:27,638 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:27,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:27,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:33:37,654 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:37,654 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:37,671 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:37,673 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:37,673 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:33:47,687 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:47,687 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:47,707 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:47,710 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:47,710 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:33:57,733 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:33:57,734 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:33:57,756 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:33:57,759 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:33:57,759 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:03,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L765 2021-12-16 05:34:03,823 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 30 seconds. Reason: Let the rebalance begin! 2021-12-16 05:34:07,772 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:34:07,772 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:34:07,792 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:07,795 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:07,795 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:17,812 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:34:17,812 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:34:17,838 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:17,842 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:17,842 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:27,868 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 05:34:27,868 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 05:34:27,888 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:27,891 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:27,891 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:33,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.23 % 2021-12-16 05:34:33,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:34:33,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:37,904 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.47 % 2021-12-16 05:34:37,904 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.47 2021-12-16 05:34:37,921 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:37,924 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:37,924 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:43,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:34:43,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:34:43,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:47,937 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:34:47,937 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:34:47,954 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:47,957 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:47,957 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:53,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:34:53,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:34:53,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:34:57,970 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:34:57,971 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:34:57,992 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:34:57,993 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:34:57,993 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:03,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:03,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:03,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:08,006 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:08,006 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:08,025 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:08,028 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:08,028 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:13,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:13,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:13,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:18,039 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:18,040 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:18,058 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:18,059 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:18,059 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:23,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:23,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:23,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:28,072 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:28,072 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:28,091 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:28,094 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:28,094 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:33,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:33,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:33,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:38,108 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:38,108 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:38,128 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:38,131 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:38,131 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:43,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:44,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:44,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:48,144 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:48,144 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:48,165 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:48,167 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:48,167 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:54,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:54,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:35:54,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:35:58,181 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.86 % 2021-12-16 05:35:58,181 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 0.86 2021-12-16 05:35:58,200 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:35:58,204 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:35:58,204 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:04,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.33 % 2021-12-16 05:36:04,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:04,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:08,217 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:08,217 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:08,236 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:08,239 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:08,239 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:14,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:14,073 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:14,073 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:18,250 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:18,252 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:18,269 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:18,273 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:18,273 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:24,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:24,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:24,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:28,286 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:28,286 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:28,303 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:28,305 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:28,305 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:34,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:34,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:34,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:38,319 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:38,319 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:38,335 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:38,336 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:38,336 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:44,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:44,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:44,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:48,367 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:48,368 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:48,387 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:48,390 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:48,390 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:54,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:54,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:36:54,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:36:58,404 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:36:58,404 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:36:58,427 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:36:58,430 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:36:58,430 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:04,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:37:04,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:04,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:08,450 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:37:08,450 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:37:08,477 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:08,482 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:08,482 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:14,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:37:14,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:14,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:18,494 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:37:18,496 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 1.72 2021-12-16 05:37:18,515 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:18,517 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:18,517 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:24,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.72 % 2021-12-16 05:37:24,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:24,213 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:28,532 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.19 % 2021-12-16 05:37:28,533 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.19 2021-12-16 05:37:28,559 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:28,561 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:28,561 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:34,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:34,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:34,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:38,588 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:38,588 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:37:38,605 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:38,608 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:38,609 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:44,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:44,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:44,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:48,621 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:48,621 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:37:48,638 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:48,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:48,641 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:54,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:54,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:37:54,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:37:58,654 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:37:58,654 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:37:58,673 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:37:58,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:37:58,677 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:04,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:04,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:04,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:08,688 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:08,688 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:38:08,707 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:08,710 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:08,710 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:14,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:14,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:14,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:18,720 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:18,721 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:38:18,743 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:18,746 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:18,746 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:24,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:24,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:24,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:28,760 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:28,760 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:38:28,780 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:28,783 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:28,785 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:34,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:34,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:34,352 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:38,809 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:38,809 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:38:38,825 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:38,828 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:38,828 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:44,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:44,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:44,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:48,842 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.58 % 2021-12-16 05:38:48,842 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 2.58 2021-12-16 05:38:48,864 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:48,868 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:48,868 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:54,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.81 % 2021-12-16 05:38:54,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:38:54,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:38:58,878 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.20 % 2021-12-16 05:38:58,880 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.2 2021-12-16 05:38:58,897 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:38:58,898 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:38:58,898 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:04,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:04,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:04,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:08,917 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:08,917 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:08,937 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:08,941 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:08,941 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:14,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:14,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:14,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:18,957 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:18,957 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:18,974 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:18,976 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:18,976 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:24,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:24,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:24,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:28,989 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:28,990 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:29,006 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:29,009 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:29,009 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:34,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:34,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:34,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:39,023 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:39,023 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:39,043 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:39,046 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:39,046 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:44,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:44,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:44,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:49,059 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:49,059 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:49,078 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:49,081 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:49,082 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:54,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:54,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:39:54,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:39:59,095 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:39:59,095 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:39:59,115 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:39:59,118 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:39:59,118 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:04,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:40:04,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:04,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:09,132 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:40:09,134 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:40:09,153 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:09,154 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:09,154 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:14,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:40:14,573 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:14,573 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:19,167 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 05:40:19,168 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 05:40:19,193 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:19,197 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:19,197 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:24,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-16 05:40:24,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:24,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:29,209 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.06 % 2021-12-16 05:40:29,209 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.06 2021-12-16 05:40:29,227 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:29,230 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:29,230 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:34,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:34,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:34,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:39,243 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:39,243 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:40:39,259 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:39,262 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:39,262 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:44,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:44,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:44,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:49,273 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:49,275 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:40:49,292 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:49,295 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:49,295 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:54,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:54,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:40:54,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:40:59,305 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:40:59,306 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:40:59,325 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:40:59,326 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:40:59,326 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:04,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:04,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:04,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:09,339 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:09,339 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:41:09,359 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:09,364 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:09,364 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:14,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:14,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:14,696 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:19,378 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:19,378 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:41:19,398 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:19,401 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:19,401 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:24,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:24,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:24,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:29,426 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:29,426 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:41:29,446 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:29,448 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:29,448 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:34,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:34,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:34,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:39,463 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:39,463 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.3 2021-12-16 05:41:39,486 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:39,490 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:39,490 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:44,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.30 % 2021-12-16 05:41:44,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:44,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:49,502 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.69 % 2021-12-16 05:41:49,502 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 4.69 2021-12-16 05:41:49,519 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:49,520 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:49,520 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:54,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.92 % 2021-12-16 05:41:54,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:41:54,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:41:59,536 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:41:59,536 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:41:59,555 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:41:59,559 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:41:59,559 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:04,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:04,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:04,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:09,572 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:09,572 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:09,592 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:09,595 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:09,595 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:14,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:14,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:14,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:19,612 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:19,612 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:19,632 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:19,635 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:19,635 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:24,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:24,839 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:24,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:29,653 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:29,653 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:29,671 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:29,674 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:29,674 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:34,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:34,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:34,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:39,687 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:39,687 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:39,704 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:39,707 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:39,707 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:44,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:44,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:44,895 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:49,720 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:49,720 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:49,736 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:49,740 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:49,740 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:54,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:54,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:42:54,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:42:59,752 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:42:59,753 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:42:59,769 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:42:59,772 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:42:59,772 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:04,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:43:04,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:04,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:09,785 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.16 % 2021-12-16 05:43:09,786 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.16 2021-12-16 05:43:09,803 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:09,806 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:09,806 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:14,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.39 % 2021-12-16 05:43:14,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:14,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:19,819 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.78 % 2021-12-16 05:43:19,819 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 5.78 2021-12-16 05:43:19,836 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:19,838 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:19,838 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:24,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.78 % 2021-12-16 05:43:24,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:24,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:29,851 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:29,851 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:43:29,868 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:29,871 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:29,871 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:35,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:35,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:35,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:39,887 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:39,887 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:43:39,907 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:39,908 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:39,908 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:45,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:45,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:45,051 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:49,921 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:49,921 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:43:49,938 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:49,940 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:49,940 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:55,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:55,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:43:55,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:43:59,966 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:43:59,966 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:43:59,989 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:43:59,993 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:43:59,993 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:05,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:44:05,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:05,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:10,009 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:44:10,009 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:44:10,028 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:44:10,029 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:44:10,029 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:15,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:44:15,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:15,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:20,045 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:44:20,046 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.02 2021-12-16 05:44:20,065 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:44:20,066 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:44:20,066 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:25,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.02 % 2021-12-16 05:44:25,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:25,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:30,081 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.17 % 2021-12-16 05:44:30,082 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.17 2021-12-16 05:44:30,098 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:44:30,101 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:44:30,101 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:35,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.17 % 2021-12-16 05:44:35,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:35,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:40,115 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.17 % 2021-12-16 05:44:40,117 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.17 2021-12-16 05:44:40,138 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:44:40,140 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:44:40,140 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:45,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.41 % 2021-12-16 05:44:45,180 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:45,180 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:50,157 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-16 05:44:50,157 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.64 2021-12-16 05:44:50,176 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:44:50,178 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:44:50,178 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:44:55,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:44:55,204 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:44:55,204 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:00,190 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:00,191 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 05:45:00,207 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:00,209 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:00,209 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:05,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:05,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:05,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:10,223 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:10,223 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 05:45:10,243 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:10,247 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:10,247 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:15,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:15,244 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:15,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:20,260 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:20,260 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 05:45:20,280 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:20,283 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:20,283 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:25,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:25,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:25,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:30,298 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:30,298 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 05:45:30,318 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:30,319 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:30,321 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:35,280 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 05:45:35,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:35,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:40,334 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:45:40,334 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-16 05:45:40,351 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:40,355 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:40,355 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:45,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:45:45,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:45,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:50,368 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:45:50,368 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-16 05:45:50,387 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:45:50,388 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:45:50,388 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:45:55,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:45:55,332 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:45:55,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:00,404 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:46:00,405 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-16 05:46:00,428 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:00,431 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:00,431 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:05,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:46:05,351 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:05,351 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:10,444 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.03 % 2021-12-16 05:46:10,444 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.03 2021-12-16 05:46:10,467 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:10,470 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:10,470 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:15,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.27 % 2021-12-16 05:46:15,369 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:15,369 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:20,483 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:20,483 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-16 05:46:20,500 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:20,503 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:20,503 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:25,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:25,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:25,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:30,519 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:30,519 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-16 05:46:30,553 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:30,558 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:30,558 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:35,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:35,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:35,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:40,572 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:40,573 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-16 05:46:40,592 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:40,594 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:40,594 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:45,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:45,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:45,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:50,608 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:50,608 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-16 05:46:50,628 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:46:50,631 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:46:50,631 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:46:55,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:46:55,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:46:55,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:00,644 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.73 % 2021-12-16 05:47:00,644 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.73 2021-12-16 05:47:00,661 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:00,664 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:00,664 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:05,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:05,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:05,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:10,678 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:10,678 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.89 2021-12-16 05:47:10,703 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:10,706 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:10,706 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:15,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:15,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:15,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:20,720 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:20,720 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.89 2021-12-16 05:47:20,739 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:20,743 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:20,743 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:25,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:25,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:25,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:30,756 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:30,756 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.89 2021-12-16 05:47:30,770 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:30,773 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:30,773 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:35,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:35,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:35,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:40,788 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.89 % 2021-12-16 05:47:40,788 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 7.89 2021-12-16 05:47:40,805 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:40,809 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:40,809 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:45,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.36 % 2021-12-16 05:47:45,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:45,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:50,826 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:47:50,828 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.59 2021-12-16 05:47:50,846 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:47:50,849 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:47:50,849 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:47:55,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:47:55,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:47:55,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:00,864 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:00,864 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.59 2021-12-16 05:48:00,888 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:00,890 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:00,891 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:05,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:05,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:05,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:10,903 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:10,903 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.59 2021-12-16 05:48:10,924 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:10,927 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:10,927 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:15,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:15,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:15,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:20,940 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:20,941 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.59 2021-12-16 05:48:20,960 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:20,963 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:20,963 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:25,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.59 % 2021-12-16 05:48:25,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:25,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:30,976 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:30,977 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 05:48:30,997 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:31,002 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:31,002 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:35,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:35,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:35,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:41,015 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:41,015 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 05:48:41,033 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:41,036 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:41,036 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:45,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:45,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:45,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:51,051 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:51,052 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 05:48:51,069 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:48:51,072 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:48:51,072 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:48:55,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:48:55,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:48:55,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:01,085 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.75 % 2021-12-16 05:49:01,085 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 8.75 2021-12-16 05:49:01,101 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:01,104 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:01,104 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:05,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.99 % 2021-12-16 05:49:05,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:05,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:11,118 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.22 % 2021-12-16 05:49:11,118 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.22 2021-12-16 05:49:11,132 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:11,135 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:11,135 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:15,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:15,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:15,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:21,151 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:21,151 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.45 2021-12-16 05:49:21,170 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:21,171 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:21,171 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:25,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:25,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:25,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:31,187 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:31,187 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.45 2021-12-16 05:49:31,206 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:31,207 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:31,207 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:35,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:35,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:35,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:41,221 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.45 % 2021-12-16 05:49:41,221 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.45 2021-12-16 05:49:41,243 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:41,246 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:41,246 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:45,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:49:45,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:45,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:51,259 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:49:51,259 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.61 2021-12-16 05:49:51,275 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:49:51,276 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:49:51,276 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:49:55,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:49:55,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:49:55,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:01,295 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:01,296 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.61 2021-12-16 05:50:01,319 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:01,322 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:01,322 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:05,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:05,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:05,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:11,334 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:11,334 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.61 2021-12-16 05:50:11,357 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:11,361 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:11,361 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:15,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:15,872 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:15,874 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:21,377 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:21,378 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.61 2021-12-16 05:50:21,401 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:21,404 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:21,404 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:25,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:25,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:25,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:31,418 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.61 % 2021-12-16 05:50:31,420 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 9.61 2021-12-16 05:50:31,440 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:31,446 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:31,446 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:35,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.08 % 2021-12-16 05:50:35,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:35,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:41,459 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:50:41,459 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-16 05:50:41,477 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:41,480 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:41,480 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:45,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:50:45,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:45,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:51,494 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:50:51,496 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-16 05:50:51,519 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:50:51,523 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:50:51,523 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:50:55,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:50:55,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:50:55,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:01,536 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:51:01,536 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.31 2021-12-16 05:51:01,555 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:01,558 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:01,558 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:05,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.31 % 2021-12-16 05:51:05,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:05,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:11,571 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:11,571 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.47 2021-12-16 05:51:11,592 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:11,594 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:11,595 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:15,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:16,005 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:16,005 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:21,607 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:21,607 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.47 2021-12-16 05:51:21,625 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:21,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:21,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:26,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:26,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:26,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:31,641 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:31,641 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.47 2021-12-16 05:51:31,657 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:31,661 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:31,661 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:36,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:36,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:36,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:41,674 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:41,676 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.47 2021-12-16 05:51:41,693 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:41,694 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:41,694 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:46,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:46,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:46,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:51,707 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:51,707 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.47 2021-12-16 05:51:51,724 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:51:51,726 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:51:51,727 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:51:56,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.47 % 2021-12-16 05:51:56,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:51:56,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:01,739 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.70 % 2021-12-16 05:52:01,740 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 10.7 2021-12-16 05:52:01,757 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:01,760 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:01,760 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:06,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.17 % 2021-12-16 05:52:06,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:06,102 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:11,773 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.17 % 2021-12-16 05:52:11,773 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.17 2021-12-16 05:52:11,795 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:11,798 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:11,798 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:16,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.17 % 2021-12-16 05:52:16,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:16,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:21,812 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.17 % 2021-12-16 05:52:21,812 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.17 2021-12-16 05:52:21,829 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:21,832 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:21,832 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:26,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:26,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:26,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:31,845 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:31,845 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:52:31,861 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:31,864 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:31,864 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:36,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:36,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:36,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:41,878 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:41,878 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:52:41,894 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:41,898 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:41,898 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:46,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:46,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:46,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:51,910 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:51,910 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:52:51,930 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:52:51,931 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:52:51,931 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:52:56,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:52:56,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:52:56,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:01,944 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:53:01,946 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:53:01,969 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:01,970 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:01,970 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:06,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:53:06,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:06,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:11,983 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:53:11,983 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:53:12,000 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:12,003 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:12,003 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:16,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:53:16,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:16,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:22,015 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.33 % 2021-12-16 05:53:22,015 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.33 2021-12-16 05:53:22,032 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:22,033 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:22,033 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:26,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.56 % 2021-12-16 05:53:26,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:26,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:32,049 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.80 % 2021-12-16 05:53:32,049 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 11.8 2021-12-16 05:53:32,075 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:32,078 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:32,078 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:36,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.03 % 2021-12-16 05:53:36,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:36,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:42,089 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.03 % 2021-12-16 05:53:42,089 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.03 2021-12-16 05:53:42,107 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:42,108 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:42,108 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:46,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.03 % 2021-12-16 05:53:46,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:46,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:52,121 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:53:52,122 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-16 05:53:52,148 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:53:52,151 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:53:52,151 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:53:56,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:53:56,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:53:56,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:02,164 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:02,164 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-16 05:54:02,181 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:02,184 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:02,184 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:06,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:06,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:06,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:12,197 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:12,197 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-16 05:54:12,213 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:12,214 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:12,214 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:16,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:16,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:16,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:22,229 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:22,229 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-16 05:54:22,250 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:22,253 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:22,253 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:26,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:26,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:26,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:32,265 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:32,266 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.19 2021-12-16 05:54:32,285 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:32,288 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:32,288 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:36,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.19 % 2021-12-16 05:54:36,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:36,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:42,299 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.42 % 2021-12-16 05:54:42,299 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.42 2021-12-16 05:54:42,316 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:42,319 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:42,319 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:46,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.42 % 2021-12-16 05:54:46,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:46,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:52,332 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.42 % 2021-12-16 05:54:52,332 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.42 2021-12-16 05:54:52,355 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:54:52,357 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:54:52,357 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:54:56,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.58 % 2021-12-16 05:54:56,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:54:56,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:02,368 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.81 % 2021-12-16 05:55:02,368 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 12.81 2021-12-16 05:55:02,384 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:02,385 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:02,385 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:06,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.81 % 2021-12-16 05:55:06,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:06,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:12,401 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:12,401 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.05 2021-12-16 05:55:12,421 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:12,426 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:12,426 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:16,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:16,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:16,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:22,437 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:22,438 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.05 2021-12-16 05:55:22,457 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:22,460 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:22,460 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:26,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:26,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:26,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:32,473 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:32,473 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.05 2021-12-16 05:55:32,494 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:32,497 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:32,497 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:36,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:36,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:36,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:42,509 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:42,509 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.05 2021-12-16 05:55:42,528 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:42,532 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:42,532 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:46,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:46,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:46,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:52,545 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:52,545 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.05 2021-12-16 05:55:52,561 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:55:52,562 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:55:52,562 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:55:56,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.05 % 2021-12-16 05:55:56,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:55:56,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:02,573 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-16 05:56:02,573 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.28 2021-12-16 05:56:02,591 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:02,592 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:02,594 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:06,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-16 05:56:06,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:06,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:12,605 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.28 % 2021-12-16 05:56:12,605 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.28 2021-12-16 05:56:12,624 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:12,625 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:12,625 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:16,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.52 % 2021-12-16 05:56:16,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:16,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:22,638 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.52 % 2021-12-16 05:56:22,638 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.52 2021-12-16 05:56:22,663 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:22,665 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:22,665 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:26,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.67 % 2021-12-16 05:56:26,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:26,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:32,677 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:32,677 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:56:32,694 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:32,700 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:32,700 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:36,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:36,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:36,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:42,711 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:42,711 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:56:42,736 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:42,737 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:42,739 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:46,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:46,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:46,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:52,752 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:52,752 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:56:52,782 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:56:52,785 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:56:52,785 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:56:56,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:56:56,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:56:56,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:02,798 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:57:02,798 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:57:02,822 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:02,825 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:02,825 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:06,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:57:06,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:06,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:12,838 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:57:12,838 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:57:12,858 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:12,861 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:12,861 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:16,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:57:16,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:16,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:22,871 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.91 % 2021-12-16 05:57:22,872 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 13.91 2021-12-16 05:57:22,890 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:22,894 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:22,894 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:26,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.14 % 2021-12-16 05:57:26,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:26,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:32,905 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.14 % 2021-12-16 05:57:32,905 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.14 2021-12-16 05:57:32,924 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:32,928 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:32,928 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:36,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.14 % 2021-12-16 05:57:36,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:36,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:42,938 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.14 % 2021-12-16 05:57:42,940 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.14 2021-12-16 05:57:42,954 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:42,957 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:42,957 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:46,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.38 % 2021-12-16 05:57:46,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:46,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:52,973 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.53 % 2021-12-16 05:57:52,973 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.53 2021-12-16 05:57:53,000 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:57:53,003 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:57:53,003 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:57:56,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:57:56,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:57:56,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:03,015 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:03,015 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.77 2021-12-16 05:58:03,030 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:03,033 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:03,033 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:06,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:06,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:06,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:13,043 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:13,045 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.77 2021-12-16 05:58:13,062 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:13,065 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:13,065 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:16,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:16,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:16,905 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:23,075 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:23,075 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.77 2021-12-16 05:58:23,094 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:23,096 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:23,096 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:26,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:26,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:26,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:33,108 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:33,108 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 14.77 2021-12-16 05:58:33,124 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:33,125 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:33,125 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:36,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.77 % 2021-12-16 05:58:36,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:36,946 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:43,138 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:58:43,138 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.0 2021-12-16 05:58:43,163 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:43,167 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:43,167 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:46,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:58:46,961 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:46,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:53,178 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:58:53,180 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.0 2021-12-16 05:58:53,197 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:58:53,200 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:58:53,200 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:58:56,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:58:56,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:58:56,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:03,213 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:59:03,213 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.0 2021-12-16 05:59:03,237 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:03,240 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:03,240 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:06,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.00 % 2021-12-16 05:59:06,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:06,999 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:13,252 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.16 % 2021-12-16 05:59:13,253 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.16 2021-12-16 05:59:13,272 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:13,275 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:13,275 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:17,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.39 % 2021-12-16 05:59:17,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:17,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:23,288 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.39 % 2021-12-16 05:59:23,288 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.39 2021-12-16 05:59:23,309 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:23,312 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:23,313 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:27,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-16 05:59:27,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:27,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:33,328 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-16 05:59:33,328 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.63 2021-12-16 05:59:33,345 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:33,346 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:33,348 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:37,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-16 05:59:37,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:37,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:43,361 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-16 05:59:43,361 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.63 2021-12-16 05:59:43,382 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:43,385 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:43,387 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:47,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.63 % 2021-12-16 05:59:47,089 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:47,089 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:53,398 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 05:59:53,398 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.86 2021-12-16 05:59:53,413 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 05:59:53,414 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 05:59:53,414 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 05:59:57,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 05:59:57,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 05:59:57,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:03,430 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:03,430 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.86 2021-12-16 06:00:03,454 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:03,470 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:03,470 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:07,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:07,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:07,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:13,483 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:13,483 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.86 2021-12-16 06:00:13,500 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:13,503 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:13,503 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:17,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:17,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:17,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:23,515 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:23,515 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.86 2021-12-16 06:00:23,532 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:23,535 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:23,535 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:27,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:27,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:27,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:33,545 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:33,545 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 15.86 2021-12-16 06:00:33,559 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:33,562 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:33,562 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:37,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.86 % 2021-12-16 06:00:37,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:37,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:43,573 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.25 % 2021-12-16 06:00:43,573 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.25 2021-12-16 06:00:43,598 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:43,601 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:43,601 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:47,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.25 % 2021-12-16 06:00:47,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:47,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:53,612 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.25 % 2021-12-16 06:00:53,614 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.25 2021-12-16 06:00:53,630 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:00:53,632 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:00:53,632 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:00:57,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.48 % 2021-12-16 06:00:57,224 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:00:57,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:03,645 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.48 % 2021-12-16 06:01:03,645 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.48 2021-12-16 06:01:03,668 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:03,671 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:03,671 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:07,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.48 % 2021-12-16 06:01:07,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:07,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:13,681 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:13,681 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:01:13,697 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:13,700 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:13,700 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:17,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:17,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:17,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:23,710 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:23,711 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:01:23,727 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:23,730 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:23,730 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:27,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:27,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:27,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:33,743 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:33,743 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:01:33,760 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:33,763 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:33,765 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:37,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:37,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:37,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:43,775 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:43,776 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:01:43,792 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:43,793 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:43,793 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:47,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:47,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:47,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:53,806 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:53,808 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:01:53,828 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:01:53,831 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:01:53,832 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:01:57,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:01:57,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:01:57,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:03,841 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:02:03,842 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 16.72 2021-12-16 06:02:03,859 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:03,864 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:03,864 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:07,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.72 % 2021-12-16 06:02:07,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:07,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:13,875 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.11 % 2021-12-16 06:02:13,875 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.11 2021-12-16 06:02:13,891 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:13,894 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:13,895 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:17,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.35 % 2021-12-16 06:02:17,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:17,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:23,907 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.35 % 2021-12-16 06:02:23,907 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.35 2021-12-16 06:02:23,923 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:23,926 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:23,927 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:27,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:27,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:27,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:33,940 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:33,940 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-16 06:02:33,967 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:33,969 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:33,970 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:37,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:37,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:37,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:43,980 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:43,980 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-16 06:02:43,996 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:44,000 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:44,000 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:47,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:47,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:47,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:54,009 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:54,010 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-16 06:02:54,026 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:02:54,028 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:02:54,029 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:02:57,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:02:57,459 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:02:57,459 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:04,042 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:03:04,042 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.58 2021-12-16 06:03:04,061 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:04,063 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:04,063 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:07,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.58 % 2021-12-16 06:03:07,487 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:07,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:14,086 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.74 % 2021-12-16 06:03:14,086 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.74 2021-12-16 06:03:14,104 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:14,107 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:14,107 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:17,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.74 % 2021-12-16 06:03:17,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:17,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:24,118 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.74 % 2021-12-16 06:03:24,118 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.74 2021-12-16 06:03:24,134 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:24,137 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:24,137 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:27,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-16 06:03:27,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:27,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:34,150 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-16 06:03:34,150 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.97 2021-12-16 06:03:34,171 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:34,174 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:34,174 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:37,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-16 06:03:37,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:37,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:44,187 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-16 06:03:44,187 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 17.97 2021-12-16 06:03:44,211 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:44,213 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:44,213 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:47,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.97 % 2021-12-16 06:03:47,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:47,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:54,224 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.21 % 2021-12-16 06:03:54,224 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.21 2021-12-16 06:03:54,242 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:03:54,244 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:03:54,246 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:03:57,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.21 % 2021-12-16 06:03:57,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:03:57,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:04,256 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-16 06:04:04,256 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-16 06:04:04,276 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:04,278 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:04,278 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:07,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-16 06:04:07,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:07,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:14,289 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-16 06:04:14,289 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-16 06:04:14,306 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:14,309 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:14,311 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:17,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-16 06:04:17,632 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:17,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:24,321 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.44 % 2021-12-16 06:04:24,321 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.44 2021-12-16 06:04:24,338 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:24,341 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:24,341 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:27,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.60 % 2021-12-16 06:04:27,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:27,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:34,354 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.60 % 2021-12-16 06:04:34,354 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.6 2021-12-16 06:04:34,381 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:34,384 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:34,384 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:37,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:04:37,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:37,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:44,398 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:04:44,398 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.83 2021-12-16 06:04:44,417 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:44,421 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:44,421 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:47,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:04:47,683 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:47,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:54,433 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:04:54,433 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.83 2021-12-16 06:04:54,450 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:04:54,451 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:04:54,451 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:04:57,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:04:57,698 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:04:57,698 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:04,463 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:05:04,464 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.83 2021-12-16 06:05:04,487 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:04,490 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:04,490 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:07,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:05:07,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:07,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:14,506 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:05:14,506 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.83 2021-12-16 06:05:14,529 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:14,532 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:14,532 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:17,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:05:17,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:17,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:24,543 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.83 % 2021-12-16 06:05:24,543 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 18.83 2021-12-16 06:05:24,562 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:24,565 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:24,565 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:27,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.07 % 2021-12-16 06:05:27,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:27,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:34,575 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.22 % 2021-12-16 06:05:34,576 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.22 2021-12-16 06:05:34,595 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:34,598 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:34,598 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:37,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:05:37,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:37,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:44,609 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:05:44,609 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:05:44,627 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:44,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:44,628 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:47,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:05:47,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:47,793 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:54,638 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:05:54,640 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:05:54,657 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:05:54,660 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:05:54,660 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:05:57,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:05:57,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:05:57,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:04,671 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:04,671 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:06:04,693 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:04,694 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:04,694 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:07,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:07,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:07,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:14,706 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:14,706 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:06:14,723 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:14,724 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:14,724 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:17,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:17,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:17,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:24,736 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:24,737 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:06:24,760 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:24,763 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:24,763 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:27,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:27,874 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:27,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:34,773 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:34,775 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:06:34,792 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:34,796 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:34,796 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:37,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:37,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:37,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:44,806 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:44,806 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.69 2021-12-16 06:06:44,829 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:44,832 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:44,832 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:47,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.69 % 2021-12-16 06:06:47,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:47,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:54,844 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 06:06:54,844 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-16 06:06:54,861 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:06:54,865 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:06:54,865 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:06:57,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.08 % 2021-12-16 06:06:57,946 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:06:57,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:07:04,880 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.32 % 2021-12-16 06:07:04,880 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 20.32 2021-12-16 06:07:04,905 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:07:04,907 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:07:04,907 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:07:07,947 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >20.08% in 1954.10899997 seconds 2021-12-16 06:07:07,973 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-16 06:07:07,973 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-16 06:07:07,974 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 06:07:08,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:07:08,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 06:07:08,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:07:08,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 06:07:08,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 06:07:08,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 06:07:08,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 06:07:08,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 06:07:08,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 06:07:08,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 06:07:08,490 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 06:07:08,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 06:07:08,549 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['265802 memcached\n'], []) 2021-12-16 06:07:08,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 06:07:08,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 06:07:08,641 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['265802 memcached\n'], []) 2021-12-16 06:07:08,642 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 06:07:08,645 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 06:07:08,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:07:08,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 06:07:09,078 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:07:09,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 06:07:09,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 06:07:09,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 06:07:09,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 06:07:09,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 06:07:09,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 06:07:09,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 06:07:09,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:07:09,246 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 06:07:09,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:07:09,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:09,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:07:09,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:07:09,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:09,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:09,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:09,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:09,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:09,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:09,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:09,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:09,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:09,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:09,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:09,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:09,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:07:09,770 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 06:07:09,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:07:09,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:09,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:07:09,805 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 06:07:09,806 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 06:07:09,809 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 06:07:09,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:07:10,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 06:07:10,267 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:07:10,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 06:07:10,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 06:07:10,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 06:07:10,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 06:07:10,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 06:07:10,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 06:07:10,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 06:07:10,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:07:10,444 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 06:07:10,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:07:10,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:10,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:10,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:10,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:10,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:10,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:10,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:10,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:10,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:10,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:10,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:10,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:10,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:10,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:11,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:11,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:11,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:11,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:11,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:11,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:11,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:11,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:11,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:11,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:11,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:11,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:11,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:11,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:11,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:11,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:11,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:11,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:12,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:12,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:12,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:12,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:12,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:12,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:07:12,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:07:12,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:07:12,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:07:12,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:07:12,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:07:12,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:07:12,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:07:12,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:07:12,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:07:12,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:12,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:07:12,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:07:12,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:07:12,868 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 06:07:12,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:07:12,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:12,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:07:12,921 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 06:07:12,921 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 06:07:12,926 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 06:07:13,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:07:13,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 06:07:13,595 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:07:13,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 06:07:13,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 06:07:13,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 06:07:13,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 06:07:13,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 06:07:13,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 06:07:13,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 06:07:13,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:07:13,756 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 06:07:13,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:07:13,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:13,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:13,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:13,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:13,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:14,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:14,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:14,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:14,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:14,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:14,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:14,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:14,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:14,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:14,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:14,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:14,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:14,920 | test | DEBUG | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.55 % 2021-12-16 06:07:14,920 | test | INFO | pool-3-thread-4 | [task:check:385] Rebalance - status: running, progress: 20.55 2021-12-16 06:07:14,937 | infra | DEBUG | pool-3-thread-4 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:07:14,940 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:07:14,940 | infra | DEBUG | pool-3-thread-4 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:07:14,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:14,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:15,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:15,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:15,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:15,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:15,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:15,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:15,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:15,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:15,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:15,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:15,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:15,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:15,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:15,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:15,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:15,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:07:15,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:07:16,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:16,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:16,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:07:16,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:07:16,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:16,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:16,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:07:16,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:07:16,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:16,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:16,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:07:16,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:07:16,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:17,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:17,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:07:17,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:07:17,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:17,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:17,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:07:17,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:07:17,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:07:17,217 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 06:07:17,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:07:17,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:17,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:07:17,267 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 06:07:17,269 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 06:07:17,272 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 06:07:17,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:07:17,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 06:07:17,815 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:07:17,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 06:07:17,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 06:07:17,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 06:07:17,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 06:07:17,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 06:07:17,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 06:07:17,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 06:07:17,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:07:17,944 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 06:07:17,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:07:17,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:07:18,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:07:18,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:18,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:07:18,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:18,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:07:18,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:18,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:07:18,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:18,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:07:18,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:18,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:07:18,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:18,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:18,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:19,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:07:19,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:19,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:07:19,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:19,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:07:19,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:19,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:07:19,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:19,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:07:19,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:19,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:07:19,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:19,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:07:19,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:19,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:07:19,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:07:19,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:19,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:07:19,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:07:19,957 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 06:07:19,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:07:19,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:07:19,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:07:19,992 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 06:07:19,992 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 06:07:19,999 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-16 06:07:19,999 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-16 06:07:24,951 | test | ERROR | pool-3-thread-4 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3a354ab07f3ac2498fd7cef4ce869574', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c9dbb15c61fcb153a3294ca735cf2436', u'status': u'notRunning'} - rebalance failed 2021-12-16 06:07:24,982 | test | INFO | pool-3-thread-4 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.67: 2021-12-16 06:07:24,982 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639907L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.907Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1216 05:26:20.063915 266035 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 06:07:24,982 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639906L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.906Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30773.215>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,489,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.15851.212>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 489}},\n infinity]}}}}}.\nRebalance Operation Id = bbbeafc05784c581f0e72afd79e026d7'} 2021-12-16 06:07:24,982 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639877L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.877Z', u'text': u'Worker <0.31696.215> (for action {move,\n {489,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.68\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.30773.215>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 489,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.15851.212>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 489}},\n infinity]}}}}'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639862L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.862Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639661589074L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:09.074Z', u'text': u'Bucket "GleamBookUsers0" rebalance appears to be swap rebalance'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1639661588225L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:08.225Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.66\' in 0 seconds.'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639661587875L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:07.875Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1639661587717L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:07.717Z', u'text': u'Deleting old data files of bucket "GleamBookUsers0"'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1639661587583L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:07.583Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"num_reader_threads">>,\n <<"num_storage_threads">>,\n <<"num_writer_threads">>,\n <<"scramsha_fallback_salt">>]'} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639661587502L, u'shortText': u'message', u'serverTime': u'2021-12-16T05:33:07.502Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = bbbeafc05784c581f0e72afd79e026d7"} 2021-12-16 06:07:24,983 | test | ERROR | pool-3-thread-4 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'3a354ab07f3ac2498fd7cef4ce869574', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=c9dbb15c61fcb153a3294ca735cf2436', u'status': u'notRunning'} - rebalance failed 2021-12-16 06:08:20,013 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-16 06:08:20,013 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:08:20,026 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-16 06:08:20,056 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-16 06:08:20,056 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:08:20,066 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:20,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:20,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:22,086 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:22,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:22,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:24,105 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:24,112 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:24,112 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:26,125 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:26,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:26,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:28,142 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:28,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:28,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:30,161 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:30,168 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:30,168 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:32,183 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:32,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:32,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:34,203 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:34,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:34,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:36,221 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:36,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:36,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:38,240 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:38,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:38,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:40,257 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:40,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:40,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:42,276 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:42,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:42,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:44,296 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:44,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:44,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:46,315 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:46,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:46,323 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:48,338 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:48,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:48,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:50,357 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:50,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:50,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:52,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:52,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:52,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:54,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:54,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:54,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:56,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:56,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:56,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:08:58,436 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:08:58,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:08:58,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:00,457 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:00,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:00,463 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:02,476 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:02,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:02,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:04,499 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:04,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:04,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:06,535 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:06,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:06,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:08,558 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:08,566 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:08,566 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:10,578 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:10,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:10,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:12,599 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:12,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:12,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:14,619 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:14,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:14,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:16,638 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:16,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:16,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:18,657 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:18,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:18,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:20,678 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:20,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:20,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:22,701 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:22,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:22,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:24,724 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:24,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:24,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:26,746 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:26,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:26,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:28,765 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:28,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:28,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:30,785 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:30,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:30,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:32,805 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:32,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:32,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:34,829 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:34,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:34,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:36,851 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:36,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:36,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:38,871 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:38,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:38,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:40,891 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:40,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:40,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:42,911 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:42,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:42,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:44,933 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:09:44,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:09:44,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:09:46,953 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-16 06:09:46,967 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-16 06:09:46,967 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:09:46,983 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-16 06:09:46,997 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-16 06:09:46,997 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:09:47,007 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-16 06:09:47,009 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 06:09:47,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:09:47,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 06:09:47,387 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:09:47,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 06:09:47,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 06:09:47,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 06:09:47,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 06:09:47,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 06:09:47,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 06:09:47,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 06:09:47,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:09:47,500 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 06:09:47,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:09:47,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:47,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:47,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:09:47,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:09:47,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:47,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:47,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:47,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:47,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:47,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:47,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:47,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:47,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:47,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:47,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:47,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:47,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:47,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:47,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:47,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:47,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:09:48,006 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 06:09:48,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:09:48,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:48,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:09:48,059 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 06:09:48,059 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 06:09:48,062 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 06:09:48,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:09:48,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 06:09:48,530 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:09:48,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 06:09:48,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 06:09:48,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 06:09:48,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 06:09:48,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 06:09:48,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 06:09:48,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 06:09:48,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:09:48,648 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 06:09:48,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:09:48,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:48,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:48,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:48,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:48,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:48,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:48,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:48,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:48,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:48,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:49,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:49,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:49,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:49,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:49,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:49,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:49,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:49,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:49,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:49,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:49,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:49,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:49,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:49,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:49,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:49,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:49,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:49,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:49,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:49,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:49,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:49,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:49,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:49,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:50,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:50,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:09:50,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:09:50,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:09:50,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:09:50,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:09:50,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:09:50,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:09:50,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:09:50,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:09:50,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:09:50,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:50,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:09:50,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:09:50,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:09:50,711 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 06:09:50,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:09:50,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:50,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:09:50,753 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 06:09:50,753 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 06:09:50,759 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 06:09:50,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:09:51,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 06:09:51,174 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:09:51,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 06:09:51,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 06:09:51,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 06:09:51,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 06:09:51,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 06:09:51,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 06:09:51,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 06:09:51,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:09:51,319 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 06:09:51,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:09:51,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:51,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:51,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:51,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:51,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:51,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:51,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:51,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:51,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:51,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:51,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:51,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:51,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:51,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:51,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:52,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:52,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:52,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:52,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:52,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:52,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:52,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:52,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:52,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:52,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:52,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:52,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:52,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:52,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:52,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:52,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:09:52,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:09:52,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:09:52,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:09:52,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:52,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:52,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:09:52,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:09:52,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:53,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:09:53,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:09:53,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:53,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:09:53,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:09:53,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:53,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:09:53,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:09:53,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:09:53,187 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 06:09:53,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:09:53,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:09:53,224 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 06:09:53,224 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 06:09:53,227 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 06:09:53,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:09:53,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 06:09:53,704 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:09:53,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 06:09:53,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 06:09:53,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 06:09:53,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 06:09:53,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 06:09:53,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 06:09:53,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 06:09:53,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:09:53,812 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 06:09:53,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:09:53,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:53,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:09:53,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:09:53,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:53,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:54,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:09:54,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:54,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:09:54,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:54,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:09:54,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:54,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:09:54,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:54,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:09:54,438 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:54,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:09:54,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:54,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:09:54,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:54,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:09:54,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:54,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:09:54,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:54,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:09:54,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:54,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:54,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:54,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:09:55,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:55,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:55,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:55,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:09:55,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:55,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:55,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:55,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:09:55,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:55,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:09:55,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:55,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:09:55,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:09:55,240 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 06:09:55,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:09:55,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:09:55,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:09:55,273 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 06:09:55,273 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 06:09:55,275 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[172.23.110.66]_OUT=[172.23.110.68]_1639661583.77 2021-12-16 06:09:55,276 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 20 2021-12-16 06:09:55,303 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639663795.28 2021-12-16 06:09:55,303 | infra | INFO | pool-3-thread-5 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639663795.28' started 2021-12-16 06:09:55,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-16 06:09:55,311 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-16 06:09:55,329 | test | DEBUG | pool-3-thread-5 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-16 06:09:55,329 | test | DEBUG | pool-3-thread-5 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-16 06:09:55,387 | test | DEBUG | pool-3-thread-5 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 06:09:55,390 | test | INFO | pool-3-thread-5 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 6.54412996148 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 16.2054676419 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 1.50862068966 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 1.83324299156 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 9.8698830205 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-16 06:09:55,398 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:09:55,398 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:09:55,414 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:09:55,417 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:09:55,417 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:05,430 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:10:05,430 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:10:05,457 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:05,461 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:05,461 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:15,473 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:10:15,474 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:10:15,493 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:15,496 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:15,496 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:25,509 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:10:25,509 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:10:25,526 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:25,529 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:25,529 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:35,542 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:10:35,543 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:10:35,563 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:35,566 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:35,566 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:45,581 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 06:10:45,581 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 06:10:45,595 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:45,598 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:45,598 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:10:55,608 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.17 % 2021-12-16 06:10:55,609 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.17 2021-12-16 06:10:55,641 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:10:55,645 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:10:55,645 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:05,658 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.50 % 2021-12-16 06:11:05,658 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.5 2021-12-16 06:11:05,676 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:05,677 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:05,677 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:15,691 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.81 % 2021-12-16 06:11:15,693 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 0.81 2021-12-16 06:11:15,714 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:15,720 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:15,720 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:25,733 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:25,734 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:11:25,755 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:25,757 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:25,757 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:35,769 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:35,769 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:11:35,789 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:35,792 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:35,792 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:45,803 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:45,805 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:11:45,828 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:45,831 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:45,831 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:55,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:55,321 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 1.15 2021-12-16 06:11:55,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:55,335 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:11:55,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:11:55,845 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:11:55,845 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:11:55,864 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:11:55,865 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:11:55,865 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:05,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:12:05,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:05,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:05,880 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:12:05,880 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:12:05,905 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:05,907 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:05,907 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:15,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:12:15,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:15,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:15,918 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:12:15,918 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.15 2021-12-16 06:12:15,936 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:15,938 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:15,940 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:25,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.15 % 2021-12-16 06:12:25,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:25,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:25,950 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.32 % 2021-12-16 06:12:25,950 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.32 2021-12-16 06:12:25,967 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:25,969 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:25,969 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:35,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.98 % 2021-12-16 06:12:35,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:35,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:35,982 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.98 % 2021-12-16 06:12:35,983 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 1.98 2021-12-16 06:12:36,005 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:36,009 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:36,010 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:45,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:12:45,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:45,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:46,023 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:12:46,025 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:12:46,049 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:46,049 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:46,051 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:55,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:12:55,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:12:55,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:12:56,063 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:12:56,063 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:12:56,092 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:12:56,094 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:12:56,094 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:05,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:05,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:05,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:06,107 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:06,108 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:13:06,125 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:06,128 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:06,128 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:15,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:15,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:15,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:16,140 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:16,141 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:13:16,164 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:16,167 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:16,167 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:25,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:25,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:25,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:26,178 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:26,178 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:13:26,200 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:26,203 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:26,203 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:35,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:35,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:35,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:36,214 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.29 % 2021-12-16 06:13:36,214 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.29 2021-12-16 06:13:36,247 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:36,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:36,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:45,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-16 06:13:45,553 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:45,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:46,263 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.62 % 2021-12-16 06:13:46,263 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.62 2021-12-16 06:13:46,289 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:46,289 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:46,290 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:55,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-16 06:13:55,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:13:55,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:13:56,301 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.79 % 2021-12-16 06:13:56,302 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 2.79 2021-12-16 06:13:56,319 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:13:56,322 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:13:56,322 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:05,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:05,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:05,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:06,335 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:06,335 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:06,357 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:06,358 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:06,358 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:15,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:15,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:15,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:16,371 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:16,371 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:16,388 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:16,391 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:16,391 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:25,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:25,640 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:25,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:26,404 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:26,404 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:26,430 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:26,433 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:26,434 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:35,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:35,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:35,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:36,446 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:36,447 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:36,469 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:36,470 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:36,470 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:45,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:45,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:45,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:46,484 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:46,484 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:46,506 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:46,510 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:46,510 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:55,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:55,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:14:55,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:14:56,525 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.44 % 2021-12-16 06:14:56,525 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 3.44 2021-12-16 06:14:56,546 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:14:56,549 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:14:56,549 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:05,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.76 % 2021-12-16 06:15:05,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:05,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:06,559 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 06:15:06,561 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-16 06:15:06,584 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:06,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:06,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:15,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 06:15:15,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:15,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:16,601 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 06:15:16,601 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.1 2021-12-16 06:15:16,618 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:16,619 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:16,619 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:25,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.27 % 2021-12-16 06:15:25,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:25,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:26,632 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.27 % 2021-12-16 06:15:26,632 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.27 2021-12-16 06:15:26,657 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:26,658 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:26,658 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:35,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.27 % 2021-12-16 06:15:35,789 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:35,789 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:36,671 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:15:36,671 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.59 2021-12-16 06:15:36,687 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:36,690 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:36,690 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:45,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:15:45,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:45,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:46,703 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:15:46,703 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.59 2021-12-16 06:15:46,729 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:46,730 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:46,730 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:55,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:15:55,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:15:55,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:15:56,742 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:15:56,742 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.59 2021-12-16 06:15:56,776 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:15:56,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:15:56,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:05,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:16:05,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:05,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:06,792 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.59 % 2021-12-16 06:16:06,792 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.59 2021-12-16 06:16:06,809 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:06,812 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:06,812 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:15,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.91 % 2021-12-16 06:16:15,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:15,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:16,826 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.91 % 2021-12-16 06:16:16,826 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 4.91 2021-12-16 06:16:16,849 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:16,852 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:16,854 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:25,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.24 % 2021-12-16 06:16:25,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:25,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:26,865 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.24 % 2021-12-16 06:16:26,865 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.24 2021-12-16 06:16:26,898 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:26,901 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:26,901 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:35,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.24 % 2021-12-16 06:16:35,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:35,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:36,914 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.24 % 2021-12-16 06:16:36,914 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.24 2021-12-16 06:16:36,940 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:36,941 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:36,943 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:45,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:16:45,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:45,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:46,954 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:16:46,954 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.73 2021-12-16 06:16:46,976 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:46,980 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:46,980 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:55,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:16:55,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:16:55,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:16:56,993 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:16:56,993 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.73 2021-12-16 06:16:57,010 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:16:57,012 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:16:57,013 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:05,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:17:05,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:05,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:07,026 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:17:07,026 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.73 2021-12-16 06:17:07,051 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:07,053 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:07,053 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:15,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:17:15,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:15,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:17,065 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.73 % 2021-12-16 06:17:17,065 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 5.73 2021-12-16 06:17:17,082 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:17,085 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:17,085 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:26,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-16 06:17:26,013 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:26,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:27,098 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.06 % 2021-12-16 06:17:27,099 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.06 2021-12-16 06:17:27,114 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:27,115 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:27,115 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:36,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-16 06:17:36,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:36,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:37,134 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-16 06:17:37,134 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.39 2021-12-16 06:17:37,151 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:37,155 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:37,155 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:46,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-16 06:17:46,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:46,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:47,165 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.39 % 2021-12-16 06:17:47,167 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.39 2021-12-16 06:17:47,187 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:47,188 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:47,188 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:56,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.56 % 2021-12-16 06:17:56,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:17:56,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:17:57,200 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.56 % 2021-12-16 06:17:57,200 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.56 2021-12-16 06:17:57,216 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:17:57,217 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:17:57,217 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:06,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:06,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:06,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:07,226 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:07,226 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 06:18:07,243 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:07,246 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:07,246 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:16,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:16,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:16,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:17,257 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:17,257 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 06:18:17,279 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:17,282 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:17,282 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:26,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:26,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:26,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:27,296 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:27,296 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 06:18:27,316 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:27,321 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:27,322 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:36,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:36,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:36,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:37,332 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:37,334 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 06:18:37,351 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:37,352 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:37,352 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:46,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:46,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:46,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:47,365 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.88 % 2021-12-16 06:18:47,365 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 6.88 2021-12-16 06:18:47,390 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:47,392 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:47,392 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:56,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.21 % 2021-12-16 06:18:56,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:18:56,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:18:57,403 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.21 % 2021-12-16 06:18:57,404 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 7.21 2021-12-16 06:18:57,421 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:18:57,424 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:18:57,424 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:06,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:06,201 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:06,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:07,436 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:07,436 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 7.71 2021-12-16 06:19:07,450 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:07,451 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:07,453 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:16,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:16,233 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:16,233 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:17,463 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:17,463 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 7.71 2021-12-16 06:19:17,480 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:17,483 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:17,484 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:26,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:26,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:26,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:27,496 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.71 % 2021-12-16 06:19:27,496 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 7.71 2021-12-16 06:19:27,516 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:27,516 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:27,517 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:36,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:36,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:36,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:37,529 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:37,529 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-16 06:19:37,552 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:37,555 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:37,555 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:46,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:46,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:46,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:47,568 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:47,568 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-16 06:19:47,588 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:47,592 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:47,592 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:56,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:56,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:19:56,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:19:57,602 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:19:57,602 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-16 06:19:57,628 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:19:57,631 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:19:57,631 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:06,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:20:06,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:06,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:07,642 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.03 % 2021-12-16 06:20:07,642 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.03 2021-12-16 06:20:07,667 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:07,674 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:07,674 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:16,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-16 06:20:16,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:16,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:17,684 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-16 06:20:17,684 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-16 06:20:17,703 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:17,706 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:17,706 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:26,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-16 06:20:26,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:26,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:27,720 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.68 % 2021-12-16 06:20:27,720 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.68 2021-12-16 06:20:27,746 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:27,749 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:27,750 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:36,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:36,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:36,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:37,763 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:37,765 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-16 06:20:37,786 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:37,789 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:37,789 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:46,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:46,411 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:46,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:47,802 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:47,802 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-16 06:20:47,821 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:47,822 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:47,822 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:56,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:56,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:20:56,430 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:20:57,835 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:20:57,836 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 8.85 2021-12-16 06:20:57,859 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:20:57,865 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:20:57,865 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:06,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.85 % 2021-12-16 06:21:06,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:06,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:07,875 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 06:21:07,875 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.17 2021-12-16 06:21:07,891 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:07,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:07,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:16,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 06:21:16,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:16,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:17,908 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 06:21:17,908 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.17 2021-12-16 06:21:17,934 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:17,937 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:17,937 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:26,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:26,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:26,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:27,951 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:27,951 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-16 06:21:27,970 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:27,973 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:27,973 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:36,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:36,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:36,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:37,984 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:37,984 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-16 06:21:38,010 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:38,013 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:38,013 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:46,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:46,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:46,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:48,023 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:48,023 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-16 06:21:48,048 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:48,051 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:48,051 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:56,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:56,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:21:56,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:21:58,063 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.83 % 2021-12-16 06:21:58,063 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 9.83 2021-12-16 06:21:58,088 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:21:58,089 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:21:58,089 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:06,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-16 06:22:06,579 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:06,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:08,105 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-16 06:22:08,105 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.0 2021-12-16 06:22:08,127 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:08,131 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:08,131 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:16,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-16 06:22:16,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:16,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:18,141 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.00 % 2021-12-16 06:22:18,142 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.0 2021-12-16 06:22:18,160 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:18,161 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:18,161 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:26,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-16 06:22:26,624 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:26,624 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:28,171 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-16 06:22:28,173 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.32 2021-12-16 06:22:28,188 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:28,190 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:28,190 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:36,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-16 06:22:36,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:36,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:38,203 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.32 % 2021-12-16 06:22:38,203 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.32 2021-12-16 06:22:38,220 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:38,221 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:38,221 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:46,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:22:46,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:46,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:48,233 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:22:48,233 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.97 2021-12-16 06:22:48,249 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:48,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:48,250 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:56,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:22:56,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:22:56,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:22:58,262 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:22:58,263 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.97 2021-12-16 06:22:58,283 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:22:58,286 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:22:58,286 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:06,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:06,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:06,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:08,296 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:08,298 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.97 2021-12-16 06:23:08,313 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:08,316 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:08,316 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:16,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:16,767 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:16,767 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:18,328 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:18,328 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.97 2021-12-16 06:23:18,351 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:18,354 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:18,354 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:26,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:26,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:26,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:28,365 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.97 % 2021-12-16 06:23:28,365 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 10.97 2021-12-16 06:23:28,381 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:28,384 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:28,384 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:36,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:36,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:36,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:38,394 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:38,394 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 11.47 2021-12-16 06:23:38,411 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:38,411 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:38,413 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:46,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:46,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:46,825 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:48,424 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:48,424 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 11.47 2021-12-16 06:23:48,447 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:48,451 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:48,451 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:56,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:56,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:23:56,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:23:58,466 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.47 % 2021-12-16 06:23:58,467 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 11.47 2021-12-16 06:23:58,490 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:23:58,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:23:58,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:06,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.80 % 2021-12-16 06:24:06,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:06,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:08,506 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.80 % 2021-12-16 06:24:08,506 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 11.8 2021-12-16 06:24:08,532 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:08,533 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:08,533 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:16,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:16,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:16,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:18,546 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:18,546 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.12 2021-12-16 06:24:18,576 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:18,578 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:18,578 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:26,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:26,915 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:26,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:28,589 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:28,589 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.12 2021-12-16 06:24:28,625 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:28,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:28,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:36,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:36,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:36,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:38,641 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.12 % 2021-12-16 06:24:38,641 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.12 2021-12-16 06:24:38,667 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:38,671 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:38,671 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:46,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:24:46,966 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:46,966 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:48,686 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:24:48,686 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.29 2021-12-16 06:24:48,703 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:48,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:48,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:56,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:24:57,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:24:57,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:24:58,719 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:24:58,720 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.29 2021-12-16 06:24:58,734 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:24:58,736 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:24:58,737 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:07,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:25:07,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:07,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:08,750 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 06:25:08,752 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.29 2021-12-16 06:25:08,776 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:08,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:08,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:17,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.93 % 2021-12-16 06:25:17,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:17,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:18,789 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.93 % 2021-12-16 06:25:18,790 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 12.93 2021-12-16 06:25:18,812 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:18,815 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:18,815 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:27,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:27,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:27,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:28,828 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:28,829 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.27 2021-12-16 06:25:28,854 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:28,858 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:28,858 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:37,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:37,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:37,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:38,871 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:38,871 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.27 2021-12-16 06:25:38,891 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:38,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:38,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:47,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:47,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:47,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:48,905 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:48,905 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.27 2021-12-16 06:25:48,931 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:48,934 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:48,934 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:57,117 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 06:25:57,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:25:57,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:25:58,944 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.44 % 2021-12-16 06:25:58,944 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.44 2021-12-16 06:25:58,963 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:25:58,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:25:58,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:07,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.44 % 2021-12-16 06:26:07,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:07,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:08,977 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.44 % 2021-12-16 06:26:08,979 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.44 2021-12-16 06:26:08,993 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:08,994 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:08,994 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:17,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.44 % 2021-12-16 06:26:17,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:17,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:19,012 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.44 % 2021-12-16 06:26:19,012 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.44 2021-12-16 06:26:19,040 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:19,045 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:19,046 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:27,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.76 % 2021-12-16 06:26:27,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:27,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:29,058 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.76 % 2021-12-16 06:26:29,059 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.76 2021-12-16 06:26:29,078 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:29,081 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:29,081 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:37,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.76 % 2021-12-16 06:26:37,213 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:37,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:39,095 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.76 % 2021-12-16 06:26:39,095 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 13.76 2021-12-16 06:26:39,131 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:39,134 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:39,134 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:47,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:26:47,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:47,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:49,147 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:26:49,147 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.41 2021-12-16 06:26:49,165 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:49,167 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:49,167 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:57,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:26:57,252 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:26:57,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:26:59,178 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:26:59,178 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.41 2021-12-16 06:26:59,193 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:26:59,194 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:26:59,194 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:07,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:07,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:07,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:09,207 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:09,207 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.41 2021-12-16 06:27:09,230 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:09,234 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:09,234 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:17,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:17,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:17,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:19,246 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:19,246 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.41 2021-12-16 06:27:19,263 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:19,266 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:19,266 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:27,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:27,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:27,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:29,279 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.41 % 2021-12-16 06:27:29,279 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.41 2021-12-16 06:27:29,305 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:29,308 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:29,308 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:37,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.59 % 2021-12-16 06:27:37,329 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:37,329 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:39,318 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.59 % 2021-12-16 06:27:39,318 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 14.59 2021-12-16 06:27:39,335 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:39,338 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:39,338 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:47,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-16 06:27:47,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:47,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:49,348 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-16 06:27:49,348 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.24 2021-12-16 06:27:49,364 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:49,365 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:49,365 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:57,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-16 06:27:57,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:27:57,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:27:59,377 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.24 % 2021-12-16 06:27:59,377 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.24 2021-12-16 06:27:59,401 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:27:59,404 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:27:59,404 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:07,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:07,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:07,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:09,414 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:09,415 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-16 06:28:09,433 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:09,437 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:09,437 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:17,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:17,400 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:17,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:19,451 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:19,453 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-16 06:28:19,474 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:19,477 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:19,477 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:27,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:27,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:27,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:29,490 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:29,490 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-16 06:28:29,507 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:29,512 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:29,512 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:37,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:37,437 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:37,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:39,523 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:39,525 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-16 06:28:39,542 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:39,545 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:39,545 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:47,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:47,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:47,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:49,558 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.56 % 2021-12-16 06:28:49,559 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 15.56 2021-12-16 06:28:49,584 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:49,585 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:49,585 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:57,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:28:57,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:28:57,476 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:28:59,599 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:28:59,599 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.38 2021-12-16 06:28:59,618 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:28:59,622 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:28:59,622 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:07,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:07,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:07,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:09,634 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:09,634 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.38 2021-12-16 06:29:09,648 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:09,651 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:09,651 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:17,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:17,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:17,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:19,663 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:19,663 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.38 2021-12-16 06:29:19,683 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:19,687 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:19,687 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:27,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:27,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:27,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:29,697 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.38 % 2021-12-16 06:29:29,697 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.38 2021-12-16 06:29:29,716 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:29,717 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:29,717 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:37,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:37,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:37,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:39,730 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:39,730 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-16 06:29:39,756 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:39,757 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:39,757 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:47,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:47,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:47,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:49,769 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:49,769 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-16 06:29:49,789 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:49,792 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:49,792 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:57,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:57,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:29:57,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:29:59,805 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.71 % 2021-12-16 06:29:59,805 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 16.71 2021-12-16 06:29:59,826 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:29:59,828 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:29:59,828 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:07,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 06:30:07,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:07,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:09,839 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 06:30:09,839 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.36 2021-12-16 06:30:09,859 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:30:09,861 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:30:09,861 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:17,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:17,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:17,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:19,875 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:19,875 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:30:19,894 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:30:19,898 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:30:19,898 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:27,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:27,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:27,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:29,911 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:29,911 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:30:29,934 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:30:29,934 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:30:29,936 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:37,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:37,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:37,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:39,947 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:39,947 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:30:39,964 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:30:39,969 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:30:39,969 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:47,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:47,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:47,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:49,980 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:49,980 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:30:49,996 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:30:49,999 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:30:49,999 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:30:57,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:30:57,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:30:57,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:00,012 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:31:00,012 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:31:00,039 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:00,040 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:00,042 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:07,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:31:07,737 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:07,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:10,056 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.53 % 2021-12-16 06:31:10,056 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 17.53 2021-12-16 06:31:10,073 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:10,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:10,076 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:17,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.19 % 2021-12-16 06:31:17,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:17,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:20,088 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.19 % 2021-12-16 06:31:20,088 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.19 2021-12-16 06:31:20,111 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:20,114 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:20,114 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:27,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.50 % 2021-12-16 06:31:27,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:27,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:30,125 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.50 % 2021-12-16 06:31:30,125 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.5 2021-12-16 06:31:30,144 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:30,145 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:30,145 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:37,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:31:37,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:37,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:40,157 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:31:40,157 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:31:40,177 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:40,180 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:40,181 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:47,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:31:47,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:47,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:50,194 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:31:50,196 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:31:50,211 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:31:50,213 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:31:50,213 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:31:57,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:31:57,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:31:57,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:00,226 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:00,226 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:32:00,250 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:00,253 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:00,253 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:07,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:07,859 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:07,859 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:10,266 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:10,266 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:32:10,283 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:10,286 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:10,286 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:17,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:17,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:17,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:20,299 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:20,301 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:32:20,316 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:20,319 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:20,319 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:27,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:27,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:27,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:30,331 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.68 % 2021-12-16 06:32:30,331 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 18.68 2021-12-16 06:32:30,361 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:30,365 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:30,365 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:37,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 06:32:37,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:37,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:40,375 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 06:32:40,375 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.32 2021-12-16 06:32:40,397 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:40,398 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:40,398 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:47,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-16 06:32:47,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:47,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:50,414 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.65 % 2021-12-16 06:32:50,414 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.65 2021-12-16 06:32:50,440 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:32:50,443 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:32:50,443 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:32:57,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:32:57,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:32:57,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:00,457 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:00,457 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.82 2021-12-16 06:33:00,480 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:00,483 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:00,483 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:07,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:07,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:07,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:10,496 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:10,497 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.82 2021-12-16 06:33:10,513 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:10,516 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:10,516 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:17,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:17,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:17,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:20,529 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:20,529 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.82 2021-12-16 06:33:20,551 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:20,553 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:20,555 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:27,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:28,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:28,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:30,566 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:30,568 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 19.82 2021-12-16 06:33:30,585 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:30,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:30,588 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:38,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.82 % 2021-12-16 06:33:38,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:38,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:40,599 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-16 06:33:40,601 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.15 2021-12-16 06:33:40,625 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:40,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:40,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:48,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-16 06:33:48,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:48,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:50,641 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-16 06:33:50,641 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.15 2021-12-16 06:33:50,671 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:33:50,674 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:33:50,674 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:33:58,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-16 06:33:58,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:33:58,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:00,687 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.15 % 2021-12-16 06:34:00,687 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.15 2021-12-16 06:34:00,703 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:00,704 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:00,704 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:08,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.48 % 2021-12-16 06:34:08,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:08,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:10,717 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.80 % 2021-12-16 06:34:10,717 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.8 2021-12-16 06:34:10,743 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:10,744 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:10,744 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:18,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.80 % 2021-12-16 06:34:18,101 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:18,101 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:20,757 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.80 % 2021-12-16 06:34:20,757 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.8 2021-12-16 06:34:20,776 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:20,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:20,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:28,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.80 % 2021-12-16 06:34:28,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:28,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:30,790 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:30,790 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-16 06:34:30,815 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:30,816 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:30,816 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:38,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:38,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:38,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:40,832 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:40,832 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-16 06:34:40,855 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:40,858 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:40,858 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:48,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:48,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:48,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:50,871 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:50,871 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-16 06:34:50,891 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:34:50,892 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:34:50,892 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:34:58,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:34:58,196 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:34:58,196 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:00,905 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:35:00,905 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-16 06:35:00,928 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:00,930 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:00,930 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:08,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:35:08,219 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:08,220 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:10,943 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.97 % 2021-12-16 06:35:10,943 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 20.97 2021-12-16 06:35:10,963 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:10,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:10,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:18,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.29 % 2021-12-16 06:35:18,242 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:18,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:20,976 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.29 % 2021-12-16 06:35:20,977 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 21.29 2021-12-16 06:35:21,000 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:21,003 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:21,003 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:28,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.29 % 2021-12-16 06:35:28,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:28,262 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:31,013 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.80 % 2021-12-16 06:35:31,013 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 21.8 2021-12-16 06:35:31,032 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:31,035 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:31,036 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:38,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:35:38,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:38,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:41,046 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:35:41,046 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:35:41,065 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:41,069 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:41,069 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:48,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:35:48,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:48,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:51,082 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:35:51,082 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:35:51,096 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:35:51,098 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:35:51,098 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:35:58,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:35:58,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:35:58,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:01,111 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:01,112 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:36:01,134 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:01,138 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:01,138 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:08,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:08,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:08,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:11,148 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:11,148 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:36:11,165 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:11,170 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:11,170 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:18,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:18,355 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:18,355 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:21,181 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:21,181 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:36:21,207 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:21,210 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:21,210 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:28,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:28,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:28,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:31,221 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.12 % 2021-12-16 06:36:31,223 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.12 2021-12-16 06:36:31,239 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:31,242 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:31,242 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:38,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 06:36:38,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:38,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:41,253 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 06:36:41,253 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.44 2021-12-16 06:36:41,273 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:41,276 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:41,276 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:48,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.76 % 2021-12-16 06:36:48,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:48,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:51,286 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.76 % 2021-12-16 06:36:51,286 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 22.76 2021-12-16 06:36:51,302 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:36:51,303 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:36:51,303 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:36:58,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-16 06:36:58,446 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:36:58,446 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:01,313 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.09 % 2021-12-16 06:37:01,313 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.09 2021-12-16 06:37:01,329 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:01,332 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:01,332 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:08,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:08,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:08,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:11,342 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:11,342 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.26 2021-12-16 06:37:11,359 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:11,362 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:11,362 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:18,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:18,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:18,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:21,377 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:21,377 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.26 2021-12-16 06:37:21,401 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:21,403 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:21,404 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:28,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:28,505 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:28,505 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:31,415 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:31,415 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.26 2021-12-16 06:37:31,437 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:31,440 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:31,440 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:38,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:38,525 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:38,525 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:41,451 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:41,453 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.26 2021-12-16 06:37:41,467 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:41,469 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:41,470 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:48,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:48,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:48,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:51,486 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.26 % 2021-12-16 06:37:51,486 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.26 2021-12-16 06:37:51,509 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:37:51,513 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:37:51,513 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:37:58,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.59 % 2021-12-16 06:37:58,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:37:58,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:01,523 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.59 % 2021-12-16 06:38:01,523 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 23.59 2021-12-16 06:38:01,539 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:01,542 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:01,542 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:08,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.24 % 2021-12-16 06:38:08,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:08,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:11,555 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.24 % 2021-12-16 06:38:11,555 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.24 2021-12-16 06:38:11,578 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:11,579 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:11,579 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:18,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.24 % 2021-12-16 06:38:18,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:18,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:21,592 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.24 % 2021-12-16 06:38:21,594 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.24 2021-12-16 06:38:21,614 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:21,618 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:21,618 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:28,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:28,627 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:28,627 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:31,628 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:31,630 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:38:31,644 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:31,645 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:31,647 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:38,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:38,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:38,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:41,661 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:41,661 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:38:41,680 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:41,684 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:41,684 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:48,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:48,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:48,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:51,694 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:51,696 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:38:51,711 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:38:51,713 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:38:51,713 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:38:58,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:38:58,688 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:38:58,688 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:01,726 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:01,726 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:39:01,747 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:01,750 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:01,750 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:08,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:08,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:08,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:11,760 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:11,760 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:39:11,785 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:11,788 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:11,788 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:18,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:18,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:18,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:21,813 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:21,813 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:39:21,829 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:21,832 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:21,832 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:28,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:28,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:28,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:31,845 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.41 % 2021-12-16 06:39:31,845 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 24.41 2021-12-16 06:39:31,867 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:31,871 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:31,871 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:38,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:39:38,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:38,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:41,881 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:39:41,882 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:39:41,911 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:41,914 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:41,914 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:48,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:39:48,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:48,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:51,926 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:39:51,926 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:39:51,950 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:39:51,953 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:39:51,953 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:39:58,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:39:58,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:39:58,815 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:01,967 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:01,969 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:40:01,999 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:02,002 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:02,002 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:08,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:08,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:08,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:12,016 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:12,016 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:40:12,036 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:12,039 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:12,039 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:18,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:18,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:18,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:22,052 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:22,053 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:40:22,076 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:22,079 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:22,079 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:28,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:28,882 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:28,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:32,091 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:32,091 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.56 2021-12-16 06:40:32,111 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:32,112 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:32,114 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:38,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 06:40:38,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:38,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:42,125 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.73 % 2021-12-16 06:40:42,125 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 25.73 2021-12-16 06:40:42,144 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:42,147 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:42,147 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:48,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.05 % 2021-12-16 06:40:48,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:48,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:52,161 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.05 % 2021-12-16 06:40:52,163 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.05 2021-12-16 06:40:52,187 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:40:52,188 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:40:52,188 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:40:58,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.39 % 2021-12-16 06:40:58,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:40:58,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:02,203 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.39 % 2021-12-16 06:41:02,203 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.39 2021-12-16 06:41:02,220 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:02,223 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:02,223 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:08,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:08,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:08,956 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:12,240 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:12,242 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:41:12,269 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:12,273 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:12,273 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:18,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:18,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:18,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:22,289 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:22,289 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:41:22,308 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:22,311 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:22,311 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:28,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:28,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:28,994 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:32,323 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:32,323 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:41:32,346 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:32,348 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:32,348 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:39,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:39,020 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:39,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:42,361 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:42,361 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:41:42,378 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:42,381 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:42,381 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:49,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:49,040 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:49,040 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:52,398 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:52,398 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:41:52,421 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:41:52,424 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:41:52,424 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:41:59,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:41:59,063 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:41:59,063 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:02,440 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.70 % 2021-12-16 06:42:02,440 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 26.7 2021-12-16 06:42:02,460 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:02,463 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:02,463 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:09,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.04 % 2021-12-16 06:42:09,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:09,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:12,476 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.04 % 2021-12-16 06:42:12,476 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.04 2021-12-16 06:42:12,493 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:12,496 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:12,496 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:19,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.53 % 2021-12-16 06:42:19,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:19,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:22,510 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.53 % 2021-12-16 06:42:22,512 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.53 2021-12-16 06:42:22,530 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:22,533 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:22,533 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:29,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:29,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:29,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:32,552 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:32,552 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 06:42:32,569 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:32,572 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:32,572 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:39,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:39,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:39,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:42,585 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:42,586 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 06:42:42,602 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:42,605 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:42,605 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:49,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:49,181 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:49,181 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:52,621 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:52,622 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 06:42:52,647 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:42:52,651 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:42:52,653 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:42:59,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:42:59,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:42:59,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:02,677 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:43:02,677 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 06:43:02,696 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:02,698 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:02,698 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:09,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:43:09,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:09,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:12,711 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:43:12,711 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 27.85 2021-12-16 06:43:12,736 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:12,737 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:12,739 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:19,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.85 % 2021-12-16 06:43:19,252 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:19,252 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:22,753 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.50 % 2021-12-16 06:43:22,753 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 28.5 2021-12-16 06:43:22,776 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:22,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:22,779 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:29,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.50 % 2021-12-16 06:43:29,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:29,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:32,789 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.50 % 2021-12-16 06:43:32,790 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 28.5 2021-12-16 06:43:32,808 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:32,809 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:32,809 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:39,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:43:39,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:39,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:42,821 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:43:42,821 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:43:42,838 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:42,842 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:42,842 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:49,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:43:49,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:49,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:52,852 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:43:52,852 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:43:52,871 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:43:52,875 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:43:52,875 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:43:59,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:43:59,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:43:59,336 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:02,887 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:02,887 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:44:02,911 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:02,913 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:02,913 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:09,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:09,352 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:09,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:12,924 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:12,924 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:44:12,943 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:12,946 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:12,946 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:19,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:19,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:19,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:22,957 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:22,957 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:44:22,973 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:22,976 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:22,976 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:29,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:29,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:29,400 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:32,987 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:32,989 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.0 2021-12-16 06:44:33,003 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:33,006 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:33,006 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:39,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.00 % 2021-12-16 06:44:39,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:39,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:43,019 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.33 % 2021-12-16 06:44:43,019 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.33 2021-12-16 06:44:43,042 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:43,046 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:43,046 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:49,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.65 % 2021-12-16 06:44:49,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:49,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:53,059 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.65 % 2021-12-16 06:44:53,059 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.65 2021-12-16 06:44:53,078 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:44:53,081 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:44:53,082 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:44:59,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.65 % 2021-12-16 06:44:59,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:44:59,469 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:03,092 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.65 % 2021-12-16 06:45:03,092 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 29.65 2021-12-16 06:45:03,114 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:03,115 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:03,115 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:09,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.82 % 2021-12-16 06:45:09,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:09,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:13,127 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:13,128 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 06:45:13,144 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:13,147 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:13,148 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:19,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:19,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:19,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:23,158 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:23,158 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 06:45:23,174 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:23,177 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:23,177 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:29,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:29,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:29,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:33,190 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:33,190 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 06:45:33,214 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:33,219 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:33,219 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:39,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:39,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:39,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:43,233 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:43,233 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 06:45:43,250 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:43,253 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:43,253 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:49,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:49,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:49,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:53,266 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.14 % 2021-12-16 06:45:53,266 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.14 2021-12-16 06:45:53,290 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:45:53,292 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:45:53,292 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:45:59,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.48 % 2021-12-16 06:45:59,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:45:59,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:03,305 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.48 % 2021-12-16 06:46:03,305 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.48 2021-12-16 06:46:03,328 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:03,331 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:03,332 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:09,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.48 % 2021-12-16 06:46:09,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:09,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:13,342 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.48 % 2021-12-16 06:46:13,342 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.48 2021-12-16 06:46:13,358 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:13,359 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:13,359 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:19,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.79 % 2021-12-16 06:46:19,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:19,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:23,371 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.79 % 2021-12-16 06:46:23,371 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 30.79 2021-12-16 06:46:23,387 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:23,388 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:23,388 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:29,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.12 % 2021-12-16 06:46:29,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:29,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:33,401 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:33,401 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.29 2021-12-16 06:46:33,418 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:33,421 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:33,421 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:39,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:39,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:39,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:43,434 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:43,434 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.29 2021-12-16 06:46:43,450 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:43,451 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:43,453 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:49,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:49,711 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:49,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:53,464 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:53,464 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.29 2021-12-16 06:46:53,490 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:46:53,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:46:53,493 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:46:59,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:46:59,739 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:46:59,739 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:03,505 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:47:03,506 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.29 2021-12-16 06:47:03,535 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:03,538 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:03,538 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:09,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:47:09,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:09,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:13,549 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.29 % 2021-12-16 06:47:13,549 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.29 2021-12-16 06:47:13,568 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:13,569 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:13,569 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:19,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 06:47:19,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:19,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:23,582 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 06:47:23,582 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.62 2021-12-16 06:47:23,601 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:23,605 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:23,605 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:29,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:29,798 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:29,798 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:33,618 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:33,618 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.94 2021-12-16 06:47:33,634 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:33,638 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:33,638 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:39,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:39,816 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:39,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:43,650 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:43,651 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.94 2021-12-16 06:47:43,665 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:43,667 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:43,667 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:49,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:49,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:49,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:53,680 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:53,680 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 31.94 2021-12-16 06:47:53,697 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:47:53,698 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:47:53,698 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:47:59,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.94 % 2021-12-16 06:47:59,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:47:59,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:03,710 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.26 % 2021-12-16 06:48:03,710 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 32.26 2021-12-16 06:48:03,729 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:03,732 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:03,732 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:09,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.44 % 2021-12-16 06:48:09,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:09,882 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:13,743 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.44 % 2021-12-16 06:48:13,743 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 32.44 2021-12-16 06:48:13,767 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:13,769 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:13,769 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:19,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.44 % 2021-12-16 06:48:19,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:19,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:23,782 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.44 % 2021-12-16 06:48:23,782 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 32.44 2021-12-16 06:48:23,805 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:23,808 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:23,809 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:29,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.77 % 2021-12-16 06:48:29,923 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:29,923 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:33,819 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.77 % 2021-12-16 06:48:33,819 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 32.77 2021-12-16 06:48:33,849 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:33,852 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:33,852 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:39,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.77 % 2021-12-16 06:48:39,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:39,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:43,867 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.77 % 2021-12-16 06:48:43,867 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 32.77 2021-12-16 06:48:43,891 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:43,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:43,894 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:49,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.09 % 2021-12-16 06:48:49,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:49,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:53,904 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.09 % 2021-12-16 06:48:53,905 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.09 2021-12-16 06:48:53,924 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:48:53,927 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:48:53,927 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:48:59,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.09 % 2021-12-16 06:48:59,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:48:59,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:03,940 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.09 % 2021-12-16 06:49:03,940 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.09 2021-12-16 06:49:03,966 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:03,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:03,967 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:10,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.41 % 2021-12-16 06:49:10,015 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:49:10,015 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:13,979 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.41 % 2021-12-16 06:49:13,979 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.41 2021-12-16 06:49:13,996 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:13,999 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:13,999 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:20,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.41 % 2021-12-16 06:49:20,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:49:20,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:24,015 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:24,015 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.58 2021-12-16 06:49:24,038 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:24,042 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:24,042 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:30,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:30,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:49:30,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:34,056 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:34,056 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.58 2021-12-16 06:49:34,081 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:34,084 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:34,084 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:40,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:40,076 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:49:40,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:44,096 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:44,098 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.58 2021-12-16 06:49:44,121 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:44,125 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:44,125 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:50,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:50,095 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:49:50,095 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:49:54,138 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:49:54,140 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.58 2021-12-16 06:49:54,158 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:49:54,161 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:49:54,161 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:00,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.58 % 2021-12-16 06:50:00,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:00,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:04,174 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.92 % 2021-12-16 06:50:04,174 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.92 2021-12-16 06:50:04,197 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:04,200 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:04,200 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:10,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.92 % 2021-12-16 06:50:10,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:10,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:14,211 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.92 % 2021-12-16 06:50:14,213 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 33.92 2021-12-16 06:50:14,229 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:14,230 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:14,230 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:20,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.23 % 2021-12-16 06:50:20,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:20,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:24,243 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.23 % 2021-12-16 06:50:24,243 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.23 2021-12-16 06:50:24,263 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:24,266 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:24,266 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:30,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.23 % 2021-12-16 06:50:30,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:30,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:34,298 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:50:34,299 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:50:34,319 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:34,321 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:34,322 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:40,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:50:40,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:40,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:44,334 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:50:44,334 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:50:44,354 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:44,357 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:44,358 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:50,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:50:50,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:50:50,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:50:54,371 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:50:54,371 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:50:54,390 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:50:54,391 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:50:54,391 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:00,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:00,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:00,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:04,408 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:04,408 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:51:04,427 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:04,434 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:04,434 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:10,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:10,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:10,273 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:14,446 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:14,446 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:51:14,463 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:14,467 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:14,467 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:20,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:20,295 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:20,295 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:24,480 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.73 % 2021-12-16 06:51:24,480 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 34.73 2021-12-16 06:51:24,503 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:24,506 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:24,506 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:30,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:30,311 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:30,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:34,519 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:34,519 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.38 2021-12-16 06:51:34,539 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:34,543 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:34,543 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:40,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:40,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:40,326 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:44,558 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:44,559 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.38 2021-12-16 06:51:44,573 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:44,575 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:44,575 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:50,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:50,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:51:50,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:51:54,589 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.38 % 2021-12-16 06:51:54,589 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.38 2021-12-16 06:51:54,608 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:51:54,611 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:51:54,611 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:00,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.70 % 2021-12-16 06:52:00,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:00,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:04,624 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.70 % 2021-12-16 06:52:04,625 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.7 2021-12-16 06:52:04,642 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:04,644 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:04,645 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:10,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:10,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:10,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:14,657 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:14,657 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.88 2021-12-16 06:52:14,688 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:14,691 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:14,691 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:20,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:20,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:20,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:24,703 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:24,703 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.88 2021-12-16 06:52:24,720 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:24,720 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:24,720 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:30,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:30,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:30,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:34,733 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:34,733 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.88 2021-12-16 06:52:34,760 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:34,763 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:34,763 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:40,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:40,438 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:40,438 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:44,776 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.88 % 2021-12-16 06:52:44,776 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 35.88 2021-12-16 06:52:44,793 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:44,796 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:44,796 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:50,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.53 % 2021-12-16 06:52:50,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:52:50,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:52:54,806 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.53 % 2021-12-16 06:52:54,808 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 36.53 2021-12-16 06:52:54,822 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:52:54,825 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:52:54,825 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:00,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.53 % 2021-12-16 06:53:00,479 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:00,479 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:04,835 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.53 % 2021-12-16 06:53:04,836 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 36.53 2021-12-16 06:53:04,852 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:04,855 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:04,855 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:10,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.53 % 2021-12-16 06:53:10,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:10,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:14,868 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.85 % 2021-12-16 06:53:14,868 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 36.85 2021-12-16 06:53:14,885 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:14,888 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:14,888 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:20,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.85 % 2021-12-16 06:53:20,528 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:20,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:24,898 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.85 % 2021-12-16 06:53:24,900 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 36.85 2021-12-16 06:53:24,921 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:24,924 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:24,926 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:30,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:30,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:30,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:34,944 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:34,946 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.02 2021-12-16 06:53:34,967 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:34,970 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:34,971 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:40,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:40,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:40,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:44,983 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:44,983 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.02 2021-12-16 06:53:45,005 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:45,007 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:45,007 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:50,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:50,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:53:50,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:53:55,023 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:53:55,023 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.02 2021-12-16 06:53:55,040 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:53:55,043 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:53:55,043 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:00,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.02 % 2021-12-16 06:54:00,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:00,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:05,058 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 06:54:05,059 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.34 2021-12-16 06:54:05,078 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:05,081 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:05,081 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:10,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.34 % 2021-12-16 06:54:10,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:10,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:15,092 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.67 % 2021-12-16 06:54:15,094 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.67 2021-12-16 06:54:15,108 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:15,109 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:15,109 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:20,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.67 % 2021-12-16 06:54:20,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:20,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:25,125 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.67 % 2021-12-16 06:54:25,127 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.67 2021-12-16 06:54:25,148 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:25,153 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:25,154 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:30,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.67 % 2021-12-16 06:54:30,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:30,681 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:35,167 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.67 % 2021-12-16 06:54:35,167 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 37.67 2021-12-16 06:54:35,191 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:35,194 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:35,194 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:40,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.00 % 2021-12-16 06:54:40,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:40,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:45,213 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.00 % 2021-12-16 06:54:45,213 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.0 2021-12-16 06:54:45,230 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:45,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:45,233 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:50,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.00 % 2021-12-16 06:54:50,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:54:50,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:54:55,246 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.00 % 2021-12-16 06:54:55,246 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.0 2021-12-16 06:54:55,273 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:54:55,276 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:54:55,278 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:00,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.17 % 2021-12-16 06:55:00,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:00,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:05,289 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.17 % 2021-12-16 06:55:05,289 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.17 2021-12-16 06:55:05,315 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:05,316 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:05,318 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:10,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.17 % 2021-12-16 06:55:10,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:10,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:15,328 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.17 % 2021-12-16 06:55:15,328 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.17 2021-12-16 06:55:15,346 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:15,349 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:15,349 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:20,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.17 % 2021-12-16 06:55:20,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:20,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:25,361 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.49 % 2021-12-16 06:55:25,361 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.49 2021-12-16 06:55:25,378 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:25,381 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:25,381 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:30,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.82 % 2021-12-16 06:55:30,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:30,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:35,391 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.82 % 2021-12-16 06:55:35,391 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.82 2021-12-16 06:55:35,410 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:35,411 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:35,413 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:40,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.82 % 2021-12-16 06:55:40,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:40,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:45,430 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.82 % 2021-12-16 06:55:45,430 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 38.82 2021-12-16 06:55:45,450 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:45,454 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:45,454 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:50,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.82 % 2021-12-16 06:55:50,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:55:50,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:55:55,467 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:55:55,467 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.14 2021-12-16 06:55:55,486 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:55:55,490 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:55:55,490 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:00,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:00,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:00,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:05,503 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:05,503 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.14 2021-12-16 06:56:05,526 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:05,528 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:05,528 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:10,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:10,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:10,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:15,539 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:15,539 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.14 2021-12-16 06:56:15,556 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:15,559 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:15,559 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:20,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:20,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:20,917 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:25,571 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:25,572 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.14 2021-12-16 06:56:25,588 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:25,589 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:25,591 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:30,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.14 % 2021-12-16 06:56:30,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:30,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:35,605 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.32 % 2021-12-16 06:56:35,605 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.32 2021-12-16 06:56:35,625 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:35,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:35,628 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:40,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.32 % 2021-12-16 06:56:40,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:40,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:45,642 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.32 % 2021-12-16 06:56:45,642 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.32 2021-12-16 06:56:45,661 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:45,665 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:45,667 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:50,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.65 % 2021-12-16 06:56:50,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:56:50,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:56:55,680 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:56:55,680 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.97 2021-12-16 06:56:55,704 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:56:55,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:56:55,707 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:01,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:01,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:57:01,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:05,717 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:05,717 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.97 2021-12-16 06:57:05,736 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:57:05,736 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:57:05,737 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:11,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:11,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:57:11,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:15,750 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:15,750 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.97 2021-12-16 06:57:15,770 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:57:15,773 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:57:15,773 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:21,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:21,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:57:21,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:25,785 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.97 % 2021-12-16 06:57:25,785 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 39.97 2021-12-16 06:57:25,805 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:57:25,809 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:57:25,809 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:31,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.29 % 2021-12-16 06:57:31,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 06:57:31,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:35,819 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.29 % 2021-12-16 06:57:35,819 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 40.29 2021-12-16 06:57:35,848 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:57:35,851 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:57:35,851 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:41,075 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >40.29% in 2745.75499988 seconds 2021-12-16 06:57:41,088 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-16 06:57:41,088 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-16 06:57:41,089 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 06:57:41,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:57:41,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 06:57:41,476 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:57:41,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 06:57:41,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 06:57:41,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 06:57:41,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 06:57:41,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 06:57:41,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 06:57:41,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 06:57:41,578 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 06:57:41,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 06:57:41,632 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['269908 memcached\n'], []) 2021-12-16 06:57:41,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 06:57:41,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 06:57:41,721 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['269908 memcached\n'], []) 2021-12-16 06:57:41,721 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 06:57:41,724 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 06:57:41,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:57:42,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 06:57:42,130 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:57:42,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 06:57:42,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 06:57:42,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 06:57:42,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 06:57:42,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 06:57:42,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 06:57:42,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 06:57:42,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:57:42,282 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 06:57:42,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:57:42,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:42,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:57:42,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:57:42,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:42,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:42,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:42,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:42,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:42,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:42,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:42,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:42,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:42,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:42,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:42,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:42,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:57:42,733 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 06:57:42,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:57:42,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:42,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:57:42,769 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 06:57:42,769 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 06:57:42,773 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 06:57:42,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:57:43,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 06:57:43,200 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:57:43,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 06:57:43,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 06:57:43,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 06:57:43,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 06:57:43,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 06:57:43,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 06:57:43,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 06:57:43,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:57:43,336 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 06:57:43,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:57:43,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:43,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:43,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:43,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:43,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:43,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:43,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:43,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:43,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:43,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:43,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:43,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:43,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:43,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:43,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:43,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:44,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:44,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:44,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:44,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:44,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:44,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:44,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:44,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:44,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:44,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:44,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:44,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:44,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:44,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:44,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:57:44,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:57:44,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:57:44,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:57:44,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:57:44,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:57:44,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:44,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:44,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:57:44,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:57:44,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:45,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:45,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:57:45,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:57:45,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:45,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:45,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:57:45,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:57:45,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:57:45,196 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 06:57:45,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:57:45,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:45,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:57:45,252 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 06:57:45,252 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 06:57:45,255 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 06:57:45,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:57:45,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 06:57:45,861 | test | DEBUG | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.29 % 2021-12-16 06:57:45,861 | test | INFO | pool-3-thread-5 | [task:check:385] Rebalance - status: running, progress: 40.29 2021-12-16 06:57:45,877 | infra | DEBUG | pool-3-thread-5 | [task:check:412] Wait before next rebalance progress check 2021-12-16 06:57:45,878 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 06:57:45,878 | infra | DEBUG | pool-3-thread-5 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 06:57:45,927 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:57:45,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 06:57:45,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 06:57:45,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 06:57:45,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 06:57:46,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 06:57:46,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 06:57:46,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 06:57:46,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:57:46,073 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 06:57:46,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:57:46,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:46,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:46,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:46,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:46,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:46,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:46,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:46,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:46,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:46,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:46,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:46,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:46,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:46,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:46,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:46,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:46,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:47,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:47,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:47,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:47,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:47,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:47,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:47,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:47,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:47,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:47,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:47,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:47,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:47,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:47,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:57:47,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 06:57:47,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:47,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:47,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:57:47,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 06:57:47,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:48,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:48,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:57:48,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 06:57:48,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:48,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:48,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:57:48,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 06:57:48,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:48,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:48,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:57:48,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 06:57:48,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:48,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:48,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:57:48,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 06:57:48,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:57:48,499 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 06:57:48,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:57:48,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:48,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:57:48,555 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 06:57:48,555 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 06:57:48,559 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 06:57:48,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 06:57:48,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 06:57:49,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 06:57:49,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 06:57:49,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 06:57:49,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 06:57:49,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 06:57:49,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 06:57:49,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 06:57:49,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 06:57:49,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 06:57:49,219 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 06:57:49,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 06:57:49,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:57:49,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 06:57:49,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:49,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 06:57:49,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:49,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 06:57:49,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:49,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 06:57:49,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:49,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 06:57:49,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:49,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:49,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:49,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 06:57:49,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:50,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 06:57:50,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:50,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 06:57:50,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:50,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 06:57:50,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:50,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 06:57:50,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:50,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 06:57:50,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:50,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 06:57:50,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:50,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 06:57:50,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:50,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 06:57:50,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 06:57:50,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:50,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 06:57:50,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 06:57:50,852 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 06:57:50,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 06:57:50,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 06:57:50,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 06:57:50,884 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 06:57:50,884 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 06:57:50,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-16 06:57:50,891 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-16 06:57:55,891 | test | ERROR | pool-3-thread-5 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c40feab86e5bd6d9ccf1e80dcac1950b', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=620cbc1fd8c98bc63f3424f769cd8477', u'status': u'notRunning'} - rebalance failed 2021-12-16 06:57:55,923 | test | INFO | pool-3-thread-5 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.67: 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673579L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.579Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.7383.221>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,454,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.10010.216>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 454}},\n infinity]}}}}}.\nRebalance Operation Id = 5c2975a567250d185da25f2820c48c9d'} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673549L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.549Z', u'text': u'Worker <0.7255.221> (for action {move,{454,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.68\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.7383.221>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 454,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.10010.216>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 454}},\n infinity]}}}}'} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673539L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.539Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1216 06:07:28.533548 270138 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673537L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.537Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663795774L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:09:55.774Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663795567L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:09:55.567Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663795334L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:09:55.334Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 5c2975a567250d185da25f2820c48c9d"} 2021-12-16 06:57:55,924 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663775467L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:09:35.467Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 115 seconds.'} 2021-12-16 06:57:55,926 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639907L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.907Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1216 05:26:20.063915 266035 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 06:57:55,926 | test | ERROR | pool-3-thread-5 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639663639906L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:07:19.906Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.30773.215>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,489,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.15851.212>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 489}},\n infinity]}}}}}.\nRebalance Operation Id = bbbeafc05784c581f0e72afd79e026d7'} 2021-12-16 06:57:55,926 | test | ERROR | pool-3-thread-5 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'c40feab86e5bd6d9ccf1e80dcac1950b', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=620cbc1fd8c98bc63f3424f769cd8477', u'status': u'notRunning'} - rebalance failed 2021-12-16 06:58:50,907 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-16 06:58:50,907 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:58:50,921 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-16 06:58:50,933 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-16 06:58:50,933 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 06:58:50,943 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:58:50,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:58:50,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:58:52,963 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:58:52,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:58:52,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:58:54,982 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:58:54,989 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:58:54,989 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:58:57,000 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:58:57,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:58:57,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:58:59,020 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:58:59,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:58:59,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:01,043 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:01,055 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:01,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:03,075 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:03,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:03,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:05,098 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:05,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:05,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:07,115 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:07,122 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:07,122 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:09,134 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:09,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:09,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:11,153 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:11,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:11,160 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:13,171 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:13,177 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:13,177 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:15,190 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:15,197 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:15,197 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:17,210 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:17,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:17,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:19,229 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:19,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:19,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:21,246 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:21,255 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:21,255 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:23,267 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:23,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:23,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:25,290 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:25,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:25,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:27,309 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:27,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:27,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:29,326 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:29,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:29,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:31,345 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:31,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:31,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:33,365 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:33,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:33,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:35,384 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:35,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:35,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:37,407 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:37,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:37,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:39,427 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:39,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:39,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:41,447 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:41,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:41,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:43,470 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:43,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:43,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:45,490 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:45,496 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:45,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:47,509 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:47,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:47,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:49,530 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:49,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:49,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:51,548 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:51,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:51,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:53,566 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:53,576 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:53,576 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:55,588 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:55,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:55,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:57,605 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:57,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:57,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 06:59:59,625 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 06:59:59,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 06:59:59,632 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:01,644 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:01,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:01,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:03,663 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:03,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:03,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:05,683 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:05,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:05,690 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:07,703 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:07,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:07,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:09,727 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:09,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:09,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:11,749 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:11,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:11,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:13,770 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:13,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:13,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:15,793 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:00:15,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:00:15,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:00:17,816 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-16 07:00:17,831 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-16 07:00:17,831 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:00:17,844 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-16 07:00:17,858 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-16 07:00:17,858 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:00:17,871 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-16 07:00:17,872 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 07:00:17,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:00:18,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 07:00:18,339 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:00:18,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 07:00:18,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 07:00:18,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 07:00:18,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 07:00:18,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 07:00:18,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 07:00:18,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 07:00:18,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:00:18,457 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 07:00:18,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:00:18,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:00:18,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:00:18,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:18,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:18,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:18,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:18,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:18,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:18,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:18,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:18,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:18,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:18,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:18,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:18,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:00:18,974 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 07:00:18,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:00:18,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:19,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:00:19,012 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 07:00:19,012 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 07:00:19,016 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 07:00:19,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:00:19,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 07:00:19,421 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:00:19,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 07:00:19,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 07:00:19,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 07:00:19,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 07:00:19,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 07:00:19,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 07:00:19,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 07:00:19,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:00:19,566 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 07:00:19,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:00:19,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:19,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:19,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:19,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:19,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:19,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:19,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:19,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:19,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:19,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:19,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:19,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:19,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:19,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:19,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:19,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:19,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:20,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:20,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:20,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:20,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:20,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:20,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:20,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:20,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:20,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:20,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:20,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:20,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:20,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:20,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:20,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:20,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:20,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:20,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:20,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:20,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:00:21,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:00:21,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:00:21,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:00:21,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:00:21,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:00:21,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:00:21,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:00:21,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:00:21,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:00:21,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:00:21,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:00:21,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:21,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:00:21,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:00:21,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:00:21,585 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 07:00:21,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:00:21,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:21,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:00:21,635 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 07:00:21,635 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 07:00:21,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 07:00:21,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:00:22,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 07:00:22,127 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:00:22,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 07:00:22,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 07:00:22,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 07:00:22,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 07:00:22,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 07:00:22,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 07:00:22,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 07:00:22,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:00:22,239 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 07:00:22,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:00:22,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:22,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:22,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:22,431 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:22,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:22,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:22,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:22,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:22,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:22,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:22,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:22,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:22,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:22,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:22,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:22,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:22,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:23,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:23,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:23,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:23,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:23,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:23,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:23,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:23,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:23,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:23,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:23,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:23,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:23,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:23,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:00:23,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:00:23,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:00:23,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:00:23,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:00:23,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:00:23,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:00:23,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:00:23,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:23,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:23,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:00:23,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:00:24,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:24,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:24,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:00:24,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:00:24,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:00:24,111 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 07:00:24,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:00:24,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:24,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:00:24,150 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 07:00:24,150 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 07:00:24,153 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 07:00:24,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:00:24,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 07:00:24,585 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:00:24,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 07:00:24,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 07:00:24,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 07:00:24,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 07:00:24,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 07:00:24,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 07:00:24,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 07:00:24,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:00:24,680 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 07:00:24,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:00:24,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:24,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:24,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:00:24,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:00:24,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:24,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:24,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:25,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:00:25,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:25,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:00:25,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:25,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:00:25,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:25,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:00:25,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:25,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:00:25,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:25,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:00:25,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:25,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:00:25,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:25,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:00:25,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:25,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:00:25,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:25,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:00:25,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:25,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:00:25,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:25,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:25,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:25,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:00:25,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:26,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:26,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:26,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:00:26,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:26,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:00:26,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:26,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:00:26,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:00:26,265 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 07:00:26,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:00:26,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:00:26,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:00:26,301 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 07:00:26,301 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 07:00:26,301 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639663795.28 2021-12-16 07:00:26,301 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 40 2021-12-16 07:00:26,329 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639666826.3 2021-12-16 07:00:26,329 | infra | INFO | pool-3-thread-6 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639666826.3' started 2021-12-16 07:00:26,336 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-16 07:00:26,336 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-16 07:00:26,357 | test | DEBUG | pool-3-thread-6 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-16 07:00:26,357 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-16 07:00:26,411 | test | DEBUG | pool-3-thread-6 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 07:00:26,414 | test | INFO | pool-3-thread-6 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 4.71260363687 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 12.9011087218 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.300100033344 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 6.87656728894 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 0.444500006945 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-16 07:00:26,424 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:00:26,424 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:00:26,441 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:00:26,444 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:00:26,444 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:00:36,454 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:00:36,454 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:00:36,474 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:00:36,477 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:00:36,477 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:00:46,490 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:00:46,490 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:00:46,507 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:00:46,509 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:00:46,509 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:00:56,522 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:00:56,522 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:00:56,546 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:00:56,548 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:00:56,548 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:06,559 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:01:06,559 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:01:06,575 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:06,578 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:06,578 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:16,591 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:01:16,592 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:01:16,615 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:16,618 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:16,618 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:26,631 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:01:26,632 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:01:26,657 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:26,661 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:26,661 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:36,673 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:01:36,674 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:01:36,694 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:36,696 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:36,696 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:46,706 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.75 % 2021-12-16 07:01:46,706 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 1.75 2021-12-16 07:01:46,720 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:46,723 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:46,723 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:01:56,734 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:01:56,734 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:01:56,753 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:01:56,756 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:01:56,756 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:06,769 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:06,769 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:06,789 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:06,792 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:06,792 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:16,805 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:16,805 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:16,826 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:16,829 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:16,829 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:26,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:26,348 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 2.49 2021-12-16 07:02:26,359 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:26,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:02:26,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:26,841 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:26,842 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:26,858 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:26,861 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:26,861 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:36,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:36,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:02:36,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:36,871 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:36,871 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:36,888 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:36,891 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:36,891 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:46,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:46,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:02:46,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:46,901 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:46,901 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:46,927 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:46,930 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:46,930 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:56,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:56,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:02:56,436 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:02:56,940 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.49 % 2021-12-16 07:02:56,940 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 2.49 2021-12-16 07:02:56,954 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:02:56,957 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:02:56,957 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:06,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.15 % 2021-12-16 07:03:06,454 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:06,454 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:06,967 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.15 % 2021-12-16 07:03:06,969 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.15 2021-12-16 07:03:06,992 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:06,993 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:06,993 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:16,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.15 % 2021-12-16 07:03:16,471 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:16,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:17,006 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.15 % 2021-12-16 07:03:17,006 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.15 2021-12-16 07:03:17,025 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:17,028 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:17,028 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:26,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:26,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:26,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:27,039 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:27,039 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.98 2021-12-16 07:03:27,056 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:27,059 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:27,059 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:36,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:36,515 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:36,515 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:37,071 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:37,072 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.98 2021-12-16 07:03:37,094 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:37,095 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:37,095 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:46,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:46,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:46,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:47,108 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:47,108 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.98 2021-12-16 07:03:47,127 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:47,128 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:47,128 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:56,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:56,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:03:56,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:03:57,141 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.98 % 2021-12-16 07:03:57,141 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 4.98 2021-12-16 07:03:57,164 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:03:57,167 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:03:57,167 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:06,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.72 % 2021-12-16 07:04:06,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:06,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:07,178 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.72 % 2021-12-16 07:04:07,178 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 5.72 2021-12-16 07:04:07,196 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:07,198 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:07,198 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:16,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.93 % 2021-12-16 07:04:16,594 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:16,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:17,209 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.93 % 2021-12-16 07:04:17,209 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 5.93 2021-12-16 07:04:17,226 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:17,230 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:17,230 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:26,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-16 07:04:26,609 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:26,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:27,243 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-16 07:04:27,243 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 6.64 2021-12-16 07:04:27,263 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:27,266 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:27,266 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:36,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-16 07:04:36,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:36,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:37,279 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.64 % 2021-12-16 07:04:37,279 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 6.64 2021-12-16 07:04:37,305 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:37,308 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:37,308 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:46,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:04:46,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:46,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:47,319 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:04:47,319 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 7.47 2021-12-16 07:04:47,335 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:47,336 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:47,338 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:56,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:04:56,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:04:56,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:04:57,348 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:04:57,349 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 7.47 2021-12-16 07:04:57,368 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:04:57,371 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:04:57,371 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:06,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:06,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:06,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:07,382 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:07,382 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 7.47 2021-12-16 07:05:07,398 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:07,400 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:07,400 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:16,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:16,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:16,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:17,413 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:17,413 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 7.47 2021-12-16 07:05:17,436 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:17,438 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:17,438 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:26,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:26,717 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:26,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:27,453 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.47 % 2021-12-16 07:05:27,453 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 7.47 2021-12-16 07:05:27,471 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:27,473 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:27,473 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:36,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-16 07:05:36,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:36,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:37,486 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.21 % 2021-12-16 07:05:37,487 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 8.21 2021-12-16 07:05:37,513 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:37,515 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:37,515 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:46,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 07:05:46,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:46,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:47,526 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 07:05:47,528 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 07:05:47,546 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:47,549 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:47,549 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:56,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 07:05:56,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:05:56,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:05:57,561 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.13 % 2021-12-16 07:05:57,561 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.13 2021-12-16 07:05:57,578 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:05:57,579 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:05:57,579 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:06,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:06,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:06,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:07,592 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:07,592 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-16 07:06:07,609 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:07,611 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:07,611 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:16,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:16,821 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:16,821 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:17,622 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:17,622 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-16 07:06:17,640 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:17,642 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:17,642 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:26,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:26,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:26,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:27,654 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:27,654 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-16 07:06:27,678 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:27,680 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:27,680 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:36,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:36,864 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:36,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:37,691 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:37,691 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-16 07:06:37,713 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:37,717 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:37,717 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:46,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:46,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:46,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:47,730 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.97 % 2021-12-16 07:06:47,730 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 9.97 2021-12-16 07:06:47,750 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:47,753 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:47,753 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:56,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-16 07:06:56,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:06:56,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:06:57,763 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.68 % 2021-12-16 07:06:57,765 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 10.68 2021-12-16 07:06:57,780 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:06:57,782 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:06:57,783 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:06,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.42 % 2021-12-16 07:07:06,917 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:06,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:07,799 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.42 % 2021-12-16 07:07:07,799 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 11.42 2021-12-16 07:07:07,821 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:07,825 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:07,825 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:16,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.62 % 2021-12-16 07:07:16,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:16,936 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:17,836 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.62 % 2021-12-16 07:07:17,836 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 11.62 2021-12-16 07:07:17,854 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:17,855 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:17,855 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:26,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:26,957 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:26,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:27,867 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:27,867 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 12.46 2021-12-16 07:07:27,882 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:27,885 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:27,885 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:36,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:36,976 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:36,976 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:37,897 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:37,897 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 12.46 2021-12-16 07:07:37,914 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:37,917 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:37,917 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:46,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:46,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:46,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:47,927 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:47,927 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 12.46 2021-12-16 07:07:47,948 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:47,950 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:47,950 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:57,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:57,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:07:57,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:07:57,963 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:07:57,963 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 12.46 2021-12-16 07:07:57,986 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:07:57,989 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:07:57,989 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:07,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:08:07,033 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:07,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:08,000 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.46 % 2021-12-16 07:08:08,000 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 12.46 2021-12-16 07:08:08,019 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:08,022 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:08,023 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:17,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.17 % 2021-12-16 07:08:17,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:17,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:18,032 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-16 07:08:18,033 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.12 2021-12-16 07:08:18,048 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:18,049 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:18,049 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:27,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-16 07:08:27,068 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:27,068 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:28,059 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-16 07:08:28,061 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.12 2021-12-16 07:08:28,078 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:28,082 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:28,082 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:37,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-16 07:08:37,085 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:37,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:38,092 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.12 % 2021-12-16 07:08:38,092 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.12 2021-12-16 07:08:38,108 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:38,111 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:38,111 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:47,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:08:47,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:47,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:48,125 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:08:48,125 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.95 2021-12-16 07:08:48,154 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:48,157 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:48,157 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:57,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:08:57,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:08:57,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:08:58,170 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:08:58,170 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.95 2021-12-16 07:08:58,187 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:08:58,191 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:08:58,191 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:07,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:07,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:07,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:08,206 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:08,207 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.95 2021-12-16 07:09:08,230 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:08,233 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:08,233 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:17,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:17,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:17,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:18,243 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:18,243 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.95 2021-12-16 07:09:18,260 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:18,263 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:18,263 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:27,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:27,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:27,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:28,276 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.95 % 2021-12-16 07:09:28,276 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 14.95 2021-12-16 07:09:28,301 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:28,302 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:28,303 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:37,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.69 % 2021-12-16 07:09:37,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:37,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:38,319 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.69 % 2021-12-16 07:09:38,321 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 15.69 2021-12-16 07:09:38,342 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:38,346 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:38,346 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:47,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:09:47,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:47,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:48,364 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:09:48,364 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 16.61 2021-12-16 07:09:48,381 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:48,385 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:48,385 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:57,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:09:57,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:09:57,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:09:58,395 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:09:58,397 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 16.61 2021-12-16 07:09:58,413 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:09:58,415 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:09:58,415 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:07,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:10:07,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:07,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:08,430 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.61 % 2021-12-16 07:10:08,430 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 16.61 2021-12-16 07:10:08,451 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:08,454 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:08,454 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:17,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:17,272 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:17,272 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:18,464 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:18,466 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 17.44 2021-12-16 07:10:18,482 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:18,483 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:18,484 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:27,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:27,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:27,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:28,500 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:28,500 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 17.44 2021-12-16 07:10:28,523 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:28,526 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:28,526 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:37,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:37,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:37,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:38,539 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:38,539 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 17.44 2021-12-16 07:10:38,559 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:38,563 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:38,563 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:47,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.44 % 2021-12-16 07:10:47,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:47,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:48,575 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-16 07:10:48,575 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 18.18 2021-12-16 07:10:48,592 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:48,594 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:48,594 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:57,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-16 07:10:57,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:10:57,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:10:58,607 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.18 % 2021-12-16 07:10:58,607 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 18.18 2021-12-16 07:10:58,631 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:10:58,634 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:10:58,634 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:07,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:07,372 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:07,372 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:08,644 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:08,645 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.1 2021-12-16 07:11:08,663 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:08,664 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:08,664 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:17,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:17,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:17,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:18,676 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:18,677 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.1 2021-12-16 07:11:18,697 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:18,700 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:18,700 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:27,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:27,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:27,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:28,713 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.10 % 2021-12-16 07:11:28,713 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.1 2021-12-16 07:11:28,736 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:28,740 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:28,740 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:37,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:37,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:37,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:38,763 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:38,763 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-16 07:11:38,780 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:38,782 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:38,782 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:47,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:47,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:47,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:48,795 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:48,795 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-16 07:11:48,811 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:48,812 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:48,813 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:57,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:57,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:11:57,473 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:11:58,825 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.93 % 2021-12-16 07:11:58,826 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 19.93 2021-12-16 07:11:58,854 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:11:58,858 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:11:58,858 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:07,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:07,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:07,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:08,871 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:08,871 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 20.65 2021-12-16 07:12:08,897 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:08,898 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:08,898 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:17,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:17,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:17,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:18,914 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:18,914 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 20.65 2021-12-16 07:12:18,940 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:18,944 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:18,944 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:27,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:27,543 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:27,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:28,956 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.65 % 2021-12-16 07:12:28,957 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 20.65 2021-12-16 07:12:28,973 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:28,976 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:28,976 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:37,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.39 % 2021-12-16 07:12:37,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:37,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:38,989 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.22 % 2021-12-16 07:12:38,990 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 22.22 2021-12-16 07:12:39,007 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:39,009 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:39,009 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:47,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:12:47,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:47,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:49,029 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:12:49,029 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-16 07:12:49,049 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:49,053 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:49,053 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:57,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:12:57,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:12:57,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:12:59,066 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:12:59,066 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-16 07:12:59,085 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:12:59,088 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:12:59,088 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:07,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:13:07,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:07,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:09,101 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.42 % 2021-12-16 07:13:09,101 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 22.42 2021-12-16 07:13:09,125 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:09,128 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:09,130 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:17,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:17,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:17,640 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:19,151 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:19,151 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 23.14 2021-12-16 07:13:19,168 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:19,171 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:19,171 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:27,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:27,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:27,663 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:29,186 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:29,186 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 23.14 2021-12-16 07:13:29,210 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:29,211 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:29,211 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:37,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:37,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:37,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:39,223 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:39,223 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 23.14 2021-12-16 07:13:39,242 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:39,243 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:39,243 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:47,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:47,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:47,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:49,257 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.14 % 2021-12-16 07:13:49,257 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 23.14 2021-12-16 07:13:49,279 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:49,280 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:49,280 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:57,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.71 % 2021-12-16 07:13:57,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:13:57,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:13:59,315 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.71 % 2021-12-16 07:13:59,315 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.71 2021-12-16 07:13:59,339 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:13:59,342 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:13:59,342 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:07,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.71 % 2021-12-16 07:14:07,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:07,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:09,357 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.71 % 2021-12-16 07:14:09,357 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.71 2021-12-16 07:14:09,374 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:09,377 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:09,377 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:17,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:17,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:17,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:19,388 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:19,388 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.91 2021-12-16 07:14:19,413 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:19,414 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:19,414 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:27,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:27,778 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:27,778 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:29,426 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:29,426 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.91 2021-12-16 07:14:29,444 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:29,447 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:29,447 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:37,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:37,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:37,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:39,460 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:39,460 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.91 2021-12-16 07:14:39,476 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:39,479 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:39,480 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:47,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:47,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:47,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:49,490 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.91 % 2021-12-16 07:14:49,490 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 24.91 2021-12-16 07:14:49,510 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:49,513 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:49,513 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:57,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.63 % 2021-12-16 07:14:57,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:14:57,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:14:59,523 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.63 % 2021-12-16 07:14:59,523 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 25.63 2021-12-16 07:14:59,540 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:14:59,543 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:14:59,543 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:07,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.37 % 2021-12-16 07:15:07,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:07,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:09,556 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.37 % 2021-12-16 07:15:09,556 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 26.37 2021-12-16 07:15:09,578 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:09,581 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:09,582 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:17,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.37 % 2021-12-16 07:15:17,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:17,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:19,592 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.20 % 2021-12-16 07:15:19,592 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.2 2021-12-16 07:15:19,608 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:19,611 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:19,611 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:27,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.20 % 2021-12-16 07:15:27,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:27,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:29,622 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.20 % 2021-12-16 07:15:29,622 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.2 2021-12-16 07:15:29,644 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:29,647 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:29,647 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:37,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:37,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:37,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:39,661 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:39,661 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.41 2021-12-16 07:15:39,681 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:39,684 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:39,684 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:47,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:47,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:47,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:49,696 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:49,696 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.41 2021-12-16 07:15:49,713 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:49,714 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:49,716 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:57,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:57,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:15:57,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:15:59,727 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:15:59,727 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.41 2021-12-16 07:15:59,752 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:15:59,753 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:15:59,753 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:07,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:16:07,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:07,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:09,765 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.41 % 2021-12-16 07:16:09,766 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 27.41 2021-12-16 07:16:09,785 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:09,788 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:09,788 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:17,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.12 % 2021-12-16 07:16:17,996 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:17,996 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:19,802 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.12 % 2021-12-16 07:16:19,802 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 28.12 2021-12-16 07:16:19,828 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:19,831 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:19,832 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:28,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.12 % 2021-12-16 07:16:28,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:28,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:29,844 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.12 % 2021-12-16 07:16:29,845 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 28.12 2021-12-16 07:16:29,869 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:29,872 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:29,872 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:38,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.12 % 2021-12-16 07:16:38,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:38,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:39,884 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.95 % 2021-12-16 07:16:39,884 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 28.95 2021-12-16 07:16:39,903 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:39,907 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:39,907 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:48,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:16:48,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:48,085 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:49,918 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:16:49,918 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 29.9 2021-12-16 07:16:49,934 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:49,936 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:49,937 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:58,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:16:58,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:16:58,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:16:59,947 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:16:59,948 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 29.9 2021-12-16 07:16:59,967 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:16:59,970 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:16:59,971 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:08,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:08,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:08,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:09,982 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:09,982 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 29.9 2021-12-16 07:17:10,000 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:17:10,003 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:17:10,003 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:18,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:18,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:18,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:20,015 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:20,016 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 29.9 2021-12-16 07:17:20,042 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:17:20,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:17:20,045 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:28,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:28,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:28,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:30,065 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.90 % 2021-12-16 07:17:30,066 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 29.9 2021-12-16 07:17:30,085 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:17:30,088 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:17:30,088 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:38,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.61 % 2021-12-16 07:17:38,198 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:38,198 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:40,098 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.61 % 2021-12-16 07:17:40,099 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 30.61 2021-12-16 07:17:40,115 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:17:40,118 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:17:40,118 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:48,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.61 % 2021-12-16 07:17:48,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:48,221 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:50,130 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.44 % 2021-12-16 07:17:50,131 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 31.44 2021-12-16 07:17:50,148 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:17:50,151 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:17:50,151 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:17:58,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.44 % 2021-12-16 07:17:58,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:17:58,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:00,161 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.44 % 2021-12-16 07:18:00,163 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 31.44 2021-12-16 07:18:00,178 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:00,181 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:00,181 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:08,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.44 % 2021-12-16 07:18:08,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:08,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:10,193 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.44 % 2021-12-16 07:18:10,194 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 31.44 2021-12-16 07:18:10,219 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:10,220 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:10,220 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:18,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:18,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:18,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:20,232 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:20,232 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 32.39 2021-12-16 07:18:20,247 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:20,250 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:20,250 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:28,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:28,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:28,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:30,260 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:30,262 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 32.39 2021-12-16 07:18:30,282 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:30,285 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:30,285 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:38,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:38,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:38,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:40,298 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:40,298 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 32.39 2021-12-16 07:18:40,321 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:40,322 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:40,323 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:48,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:48,339 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:48,339 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:50,334 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.39 % 2021-12-16 07:18:50,334 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 32.39 2021-12-16 07:18:50,351 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:18:50,354 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:18:50,354 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:18:58,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.10 % 2021-12-16 07:18:58,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:18:58,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:00,365 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.10 % 2021-12-16 07:19:00,365 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 33.1 2021-12-16 07:19:00,384 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:00,387 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:00,387 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:08,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.10 % 2021-12-16 07:19:08,382 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:08,382 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:10,398 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.10 % 2021-12-16 07:19:10,398 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 33.1 2021-12-16 07:19:10,415 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:10,420 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:10,420 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:18,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 07:19:18,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:18,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:20,430 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 07:19:20,431 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 33.94 2021-12-16 07:19:20,456 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:20,457 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:20,457 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:28,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 07:19:28,426 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:28,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:30,470 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 07:19:30,470 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 33.94 2021-12-16 07:19:30,494 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:30,497 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:30,497 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:38,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.94 % 2021-12-16 07:19:38,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:38,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:40,509 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.14 % 2021-12-16 07:19:40,509 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 34.14 2021-12-16 07:19:40,526 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:40,529 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:40,529 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:48,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.14 % 2021-12-16 07:19:48,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:48,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:50,540 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.14 % 2021-12-16 07:19:50,540 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 34.14 2021-12-16 07:19:50,559 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:19:50,562 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:19:50,562 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:19:58,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.88 % 2021-12-16 07:19:58,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:19:58,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:00,572 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:00,572 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-16 07:20:00,589 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:00,592 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:00,592 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:08,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:08,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:08,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:10,604 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:10,604 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-16 07:20:10,624 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:10,625 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:10,625 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:18,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:18,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:18,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:20,641 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:20,641 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 35.59 2021-12-16 07:20:20,664 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:20,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:20,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:28,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.59 % 2021-12-16 07:20:28,563 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:28,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:30,680 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:30,680 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 36.43 2021-12-16 07:20:30,697 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:30,700 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:30,700 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:38,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:38,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:38,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:40,711 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:40,711 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 36.43 2021-12-16 07:20:40,736 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:40,739 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:40,739 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:48,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:48,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:48,596 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:50,752 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:50,753 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 36.43 2021-12-16 07:20:50,770 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:20:50,773 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:20:50,775 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:20:58,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:20:58,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:20:58,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:00,789 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:21:00,789 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 36.43 2021-12-16 07:21:00,805 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:00,806 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:00,806 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:08,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:21:08,638 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:08,638 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:10,819 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.43 % 2021-12-16 07:21:10,819 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 36.43 2021-12-16 07:21:10,838 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:10,842 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:10,842 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:18,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.37 % 2021-12-16 07:21:18,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:18,657 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:20,852 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:20,852 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.09 2021-12-16 07:21:20,868 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:20,871 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:20,871 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:28,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:28,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:28,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:30,887 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:30,888 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.09 2021-12-16 07:21:30,911 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:30,915 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:30,915 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:38,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:38,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:38,693 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:40,927 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:40,927 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.09 2021-12-16 07:21:40,944 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:40,947 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:40,947 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:48,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:48,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:48,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:50,957 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:50,959 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.09 2021-12-16 07:21:50,974 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:21:50,976 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:21:50,976 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:21:58,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:21:58,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:21:58,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:00,990 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:22:00,992 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.09 2021-12-16 07:22:01,010 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:01,015 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:01,015 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:08,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.09 % 2021-12-16 07:22:08,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:08,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:11,026 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.92 % 2021-12-16 07:22:11,026 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.92 2021-12-16 07:22:11,045 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:11,046 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:11,046 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:18,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.92 % 2021-12-16 07:22:18,762 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:18,762 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:21,059 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.92 % 2021-12-16 07:22:21,061 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 38.92 2021-12-16 07:22:21,079 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:21,084 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:21,084 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:28,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.92 % 2021-12-16 07:22:28,795 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:28,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:31,098 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.63 % 2021-12-16 07:22:31,099 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 39.63 2021-12-16 07:22:31,115 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:31,119 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:31,119 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:38,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.84 % 2021-12-16 07:22:38,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:38,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:41,131 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.84 % 2021-12-16 07:22:41,131 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 39.84 2021-12-16 07:22:41,158 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:41,161 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:41,161 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:48,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:22:48,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:48,835 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:51,177 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:22:51,177 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 40.58 2021-12-16 07:22:51,197 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:22:51,200 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:22:51,200 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:22:58,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:22:58,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:22:58,854 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:01,211 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:01,211 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 40.58 2021-12-16 07:23:01,236 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:01,237 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:01,237 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:08,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:08,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:08,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:11,252 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:11,252 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 40.58 2021-12-16 07:23:11,269 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:11,273 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:11,273 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:18,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:18,894 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:18,894 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:21,285 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:21,285 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 40.58 2021-12-16 07:23:21,302 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:21,305 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:21,305 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:28,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:28,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:28,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:31,318 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.58 % 2021-12-16 07:23:31,318 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 40.58 2021-12-16 07:23:31,344 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:31,345 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:31,345 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:38,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.41 % 2021-12-16 07:23:38,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:38,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:41,358 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.41 % 2021-12-16 07:23:41,358 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 41.41 2021-12-16 07:23:41,378 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:41,381 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:41,381 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:48,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.61 % 2021-12-16 07:23:48,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:48,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:51,392 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.61 % 2021-12-16 07:23:51,392 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 41.61 2021-12-16 07:23:51,408 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:23:51,411 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:23:51,411 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:23:58,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.61 % 2021-12-16 07:23:58,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:23:58,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:01,428 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.35 % 2021-12-16 07:24:01,428 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 42.35 2021-12-16 07:24:01,451 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:01,454 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:01,456 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:08,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:08,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:08,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:11,467 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:11,467 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 43.07 2021-12-16 07:24:11,483 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:11,484 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:11,484 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:19,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:19,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:19,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:21,496 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:21,496 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 43.07 2021-12-16 07:24:21,515 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:21,516 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:21,516 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:29,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:29,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:29,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:31,528 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:31,529 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 43.07 2021-12-16 07:24:31,545 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:31,548 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:31,548 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:39,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:39,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:39,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:41,559 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:41,559 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 43.07 2021-12-16 07:24:41,576 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:41,578 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:41,578 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:49,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:49,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:49,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:51,595 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.07 % 2021-12-16 07:24:51,595 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 43.07 2021-12-16 07:24:51,637 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:24:51,640 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:24:51,640 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:24:59,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.90 % 2021-12-16 07:24:59,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:24:59,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:01,651 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.10 % 2021-12-16 07:25:01,651 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 44.1 2021-12-16 07:25:01,667 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:01,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:01,670 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:09,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.10 % 2021-12-16 07:25:09,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:09,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:11,680 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.10 % 2021-12-16 07:25:11,680 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 44.1 2021-12-16 07:25:11,703 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:11,707 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:11,707 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:19,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.10 % 2021-12-16 07:25:19,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:19,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:21,717 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.10 % 2021-12-16 07:25:21,717 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 44.1 2021-12-16 07:25:21,733 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:21,734 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:21,736 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:29,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:29,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:29,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:31,746 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:31,746 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:25:31,766 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:31,769 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:31,769 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:39,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:39,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:39,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:41,779 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:41,779 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:25:41,802 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:41,805 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:41,805 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:49,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:49,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:49,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:51,821 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:51,821 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:25:51,842 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:25:51,845 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:25:51,845 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:25:59,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:25:59,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:25:59,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:01,859 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:26:01,859 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:26:01,884 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:01,888 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:01,888 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:09,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:26:09,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:09,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:11,903 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:26:11,903 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:26:11,921 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:11,924 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:11,924 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:19,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:26:19,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:19,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:21,937 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.56 % 2021-12-16 07:26:21,938 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 45.56 2021-12-16 07:26:21,963 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:21,964 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:21,964 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:29,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.60 % 2021-12-16 07:26:29,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:29,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:31,977 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.60 % 2021-12-16 07:26:31,977 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 46.6 2021-12-16 07:26:31,996 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:31,999 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:31,999 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:39,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.60 % 2021-12-16 07:26:39,306 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:39,306 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:42,010 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.60 % 2021-12-16 07:26:42,010 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 46.6 2021-12-16 07:26:42,033 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:42,036 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:42,036 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:49,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.34 % 2021-12-16 07:26:49,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:49,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:52,052 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:26:52,052 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.05 2021-12-16 07:26:52,075 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:26:52,079 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:26:52,079 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:26:59,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:26:59,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:26:59,341 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:02,092 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:02,092 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.05 2021-12-16 07:27:02,108 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:02,111 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:02,111 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:09,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:09,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:09,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:12,124 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:12,124 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.05 2021-12-16 07:27:12,148 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:12,150 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:12,150 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:19,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:19,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:19,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:22,161 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:22,161 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.05 2021-12-16 07:27:22,177 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:22,180 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:22,180 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:29,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:29,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:29,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:32,194 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.05 % 2021-12-16 07:27:32,194 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.05 2021-12-16 07:27:32,219 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:32,220 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:32,220 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:39,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.25 % 2021-12-16 07:27:39,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:39,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:42,237 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.25 % 2021-12-16 07:27:42,237 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 48.25 2021-12-16 07:27:42,256 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:42,260 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:42,260 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:49,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:27:49,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:49,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:52,272 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:27:52,273 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 49.09 2021-12-16 07:27:52,289 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:27:52,290 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:27:52,290 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:27:59,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:27:59,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:27:59,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:02,303 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:28:02,303 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 49.09 2021-12-16 07:28:02,325 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:02,326 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:02,326 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:09,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:28:09,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:09,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:12,338 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.09 % 2021-12-16 07:28:12,338 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 49.09 2021-12-16 07:28:12,355 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:12,358 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:12,358 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:19,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:19,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:19,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:22,369 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:22,369 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 50.54 2021-12-16 07:28:22,391 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:22,392 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:22,392 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:29,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:29,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:29,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:32,404 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:32,404 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 50.54 2021-12-16 07:28:32,426 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:32,428 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:32,428 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:39,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:39,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:39,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:42,438 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:42,438 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 50.54 2021-12-16 07:28:42,456 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:42,457 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:42,457 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:49,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:49,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:49,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:52,470 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.54 % 2021-12-16 07:28:52,470 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 50.54 2021-12-16 07:28:52,487 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:28:52,490 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:28:52,490 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:28:59,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.75 % 2021-12-16 07:28:59,586 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:28:59,586 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:02,500 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.75 % 2021-12-16 07:29:02,502 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 50.75 2021-12-16 07:29:02,519 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:02,523 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:02,523 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:09,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.75 % 2021-12-16 07:29:09,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:09,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:12,535 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.58 % 2021-12-16 07:29:12,536 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 51.58 2021-12-16 07:29:12,561 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:12,562 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:12,562 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:19,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.58 % 2021-12-16 07:29:19,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:19,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:22,575 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.58 % 2021-12-16 07:29:22,576 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 51.58 2021-12-16 07:29:22,596 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:22,599 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:22,599 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:29,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.58 % 2021-12-16 07:29:29,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:29,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:32,611 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.32 % 2021-12-16 07:29:32,611 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 52.32 2021-12-16 07:29:32,628 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:32,631 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:32,631 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:39,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:29:39,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:39,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:42,642 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:29:42,642 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:29:42,667 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:42,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:42,668 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:49,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:29:49,683 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:49,683 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:52,680 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:29:52,680 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:29:52,704 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:29:52,707 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:29:52,707 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:29:59,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:29:59,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:29:59,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:02,720 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:02,720 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:30:02,737 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:02,740 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:02,740 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:09,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:09,721 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:09,721 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:12,752 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:12,752 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:30:12,770 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:12,773 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:12,773 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:19,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:19,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:19,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:22,783 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:22,783 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:30:22,801 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:22,802 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:22,802 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:29,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:29,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:29,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:32,813 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.03 % 2021-12-16 07:30:32,815 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.03 2021-12-16 07:30:32,834 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:32,835 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:32,835 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:39,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 07:30:39,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:39,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:42,848 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 07:30:42,849 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 07:30:42,894 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:42,898 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:42,898 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:49,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.81 % 2021-12-16 07:30:49,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:49,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:52,911 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:30:52,911 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:30:52,928 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:30:52,931 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:30:52,931 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:30:59,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:30:59,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:30:59,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:02,944 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:02,946 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:02,971 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:02,973 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:02,973 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:09,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:09,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:09,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:12,986 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:12,986 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:13,003 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:13,006 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:13,006 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:19,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:19,868 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:19,868 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:23,019 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:23,019 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:23,040 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:23,043 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:23,043 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:29,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:29,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:29,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:33,055 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:33,055 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:33,075 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:33,078 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:33,078 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:39,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:39,908 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:39,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:43,088 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:43,088 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:43,105 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:43,107 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:43,107 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:49,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:49,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:49,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:53,118 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:53,118 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 55.53 2021-12-16 07:31:53,137 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:31:53,140 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:31:53,141 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:31:59,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.53 % 2021-12-16 07:31:59,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:31:59,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:03,151 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.36 % 2021-12-16 07:32:03,151 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 56.36 2021-12-16 07:32:03,168 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:03,171 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:03,171 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:09,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.56 % 2021-12-16 07:32:09,971 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:32:09,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:13,184 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:13,184 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:32:13,220 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:13,221 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:13,223 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:19,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:19,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:32:19,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:23,233 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:23,233 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:32:23,250 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:23,253 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:23,253 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:30,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:30,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:32:30,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:33,263 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:33,263 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:32:33,286 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:33,289 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:33,289 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:40,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:40,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:32:40,032 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:43,299 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:43,299 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:32:43,318 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:43,319 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:43,321 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:50,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:50,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:32:50,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:32:53,332 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:32:53,332 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:32:53,349 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:32:53,351 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:32:53,351 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:00,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:33:00,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:00,076 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:03,367 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:33:03,367 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:33:03,391 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:03,392 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:03,394 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:10,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:33:10,098 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:10,098 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:13,405 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.02 % 2021-12-16 07:33:13,407 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 58.02 2021-12-16 07:33:13,427 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:13,428 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:13,428 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:20,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.85 % 2021-12-16 07:33:20,114 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:20,114 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:23,440 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.05 % 2021-12-16 07:33:23,440 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 59.05 2021-12-16 07:33:23,459 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:23,460 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:23,460 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:30,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.05 % 2021-12-16 07:33:30,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:30,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:33,473 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.79 % 2021-12-16 07:33:33,473 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 59.79 2021-12-16 07:33:33,492 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:33,496 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:33,496 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:40,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.79 % 2021-12-16 07:33:40,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:40,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:43,507 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.79 % 2021-12-16 07:33:43,507 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 59.79 2021-12-16 07:33:43,529 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:43,532 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:43,532 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:50,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.51 % 2021-12-16 07:33:50,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:33:50,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:33:53,546 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.51 % 2021-12-16 07:33:53,546 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 60.51 2021-12-16 07:33:53,565 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:33:53,568 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:33:53,569 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:34:00,174 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >60.51% in 1893.82599998 seconds 2021-12-16 07:34:00,187 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: true command: ns_orchestrator:needs_rebalance() 2021-12-16 07:34:00,187 | test | INFO | MainThread | [opd:abort_rebalance:779] Abort rebalance 2021-12-16 07:34:00,188 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 07:34:00,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:34:00,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 07:34:00,516 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:34:00,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 07:34:00,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 07:34:00,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 07:34:00,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 07:34:00,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 07:34:00,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 07:34:00,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 07:34:00,658 | infra | DEBUG | MainThread | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 07:34:00,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 07:34:00,711 | infra | DEBUG | MainThread | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['275380 memcached\n'], []) 2021-12-16 07:34:00,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 07:34:00,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 07:34:00,799 | infra | DEBUG | MainThread | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['275380 memcached\n'], []) 2021-12-16 07:34:00,799 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 07:34:00,802 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 07:34:00,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:34:01,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 07:34:01,315 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:34:01,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 07:34:01,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 07:34:01,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 07:34:01,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 07:34:01,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 07:34:01,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 07:34:01,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 07:34:01,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:34:01,433 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 07:34:01,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:34:01,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:01,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:01,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:34:01,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:34:01,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:01,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:01,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:01,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:01,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:01,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:01,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:01,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:01,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:01,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:01,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:01,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:01,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:01,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:02,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:02,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:02,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:02,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:02,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:02,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:02,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:34:02,207 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 07:34:02,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:34:02,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:02,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:34:02,265 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 07:34:02,266 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 07:34:02,269 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 07:34:02,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:34:02,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 07:34:02,717 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:34:02,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 07:34:02,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 07:34:02,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 07:34:02,775 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 07:34:02,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 07:34:02,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 07:34:02,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 07:34:02,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:34:02,849 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 07:34:02,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:34:02,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:02,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:02,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:03,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:03,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:03,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:03,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:03,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:03,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:03,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:03,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:03,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:03,579 | test | DEBUG | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.51 % 2021-12-16 07:34:03,579 | test | INFO | pool-3-thread-6 | [task:check:385] Rebalance - status: running, progress: 60.51 2021-12-16 07:34:03,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,605 | infra | DEBUG | pool-3-thread-6 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:34:03,608 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:34:03,608 | infra | DEBUG | pool-3-thread-6 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:34:03,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:03,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:03,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:03,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:03,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:03,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:03,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:04,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:04,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:04,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:04,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:04,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:04,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:04,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:04,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:04,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:04,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:04,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:04,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:04,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:04,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:04,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:04,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:04,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:04,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:34:05,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:34:05,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:34:05,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:34:05,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:34:05,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:34:05,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:34:05,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:34:05,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:34:05,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:34:05,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:34:05,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:34:05,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:05,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:05,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:34:05,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:34:05,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:34:06,009 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 07:34:06,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:34:06,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:06,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:34:06,075 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 07:34:06,075 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 07:34:06,078 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 07:34:06,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:34:06,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 07:34:06,621 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:34:06,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 07:34:06,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 07:34:06,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 07:34:06,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 07:34:06,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 07:34:06,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 07:34:06,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 07:34:06,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:34:06,776 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 07:34:06,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:34:06,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:06,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:06,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:06,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:06,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:06,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:07,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:07,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:07,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:07,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:07,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:07,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:07,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:07,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:07,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:07,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:07,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:07,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:07,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:07,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:07,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:07,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:07,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:07,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:07,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:07,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:08,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:08,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:08,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:08,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:08,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:08,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:08,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:08,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:34:08,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:34:08,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:34:08,737 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:34:08,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:34:08,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:34:08,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:08,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:08,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:34:09,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:34:09,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:09,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:09,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:34:09,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:34:09,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:09,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:09,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:34:09,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:34:09,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:34:09,259 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 07:34:09,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:34:09,270 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:09,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:34:09,298 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 07:34:09,298 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 07:34:09,301 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 07:34:09,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:34:09,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 07:34:09,782 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:34:09,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 07:34:09,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 07:34:09,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 07:34:09,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 07:34:09,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 07:34:09,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 07:34:09,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 07:34:09,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:34:09,901 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 07:34:09,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:34:09,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:09,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:34:10,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:34:10,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:10,186 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:34:10,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:10,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:34:10,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:10,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:34:10,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:10,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:34:10,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:10,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:34:10,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:10,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:34:10,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,785 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:10,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:34:10,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:10,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:34:10,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:10,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:10,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:11,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:34:11,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:11,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:11,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:34:11,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:11,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:11,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:34:11,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:11,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:11,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:34:11,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:11,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:11,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:34:11,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:34:11,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:11,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:34:11,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:34:11,641 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 07:34:11,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:34:11,653 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:34:11,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:34:11,677 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 07:34:11,677 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 07:34:11,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L788 2021-12-16 07:34:11,684 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 60 seconds. Reason: Sleep after error introduction 2021-12-16 07:34:13,618 | test | ERROR | pool-3-thread-6 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'4790835d4e7dbb6e01b4756829b0ead7', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=1c96663bcfa9be73e7f23a83601e8116', u'status': u'notRunning'} - rebalance failed 2021-12-16 07:34:13,655 | test | INFO | pool-3-thread-6 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.67: 2021-12-16 07:34:13,655 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639668853045L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:34:13.045Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.31880.224>,\n {{{{nocatch,{error,closed}},\n [{mc_binary,recv_with_data,4,\n [{file,"src/mc_binary.erl"},{line,41}]},\n {mc_binary,quick_active_recv,3,\n [{file,"src/mc_binary.erl"},{line,48}]},\n {mc_binary,quick_stats_loop_enter,5,\n [{file,"src/mc_binary.erl"},{line,100}]},\n {mc_binary,quick_stats,5,\n [{file,"src/mc_binary.erl"},{line,85}]},\n {mc_client_binary,get_dcp_docs_estimate,\n 3,\n [{file,"src/mc_client_binary.erl"},\n {line,754}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,616}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,225}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,226}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,430,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.16910.221>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 430}},\n infinity]}}}}}.\nRebalance Operation Id = 01d80168876166fe017db5c41d3a9b1b'} 2021-12-16 07:34:13,655 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639668853001L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:34:13.001Z', u'text': u'Worker <0.31792.224> (for action {move,\n {430,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.68\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.31880.224>,\n {{{{nocatch,\n {error,\n closed}},\n [{mc_binary,\n recv_with_data,\n 4,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 41}]},\n {mc_binary,\n quick_active_recv,\n 3,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 48}]},\n {mc_binary,\n quick_stats_loop_enter,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 100}]},\n {mc_binary,\n quick_stats,\n 5,\n [{file,\n "src/mc_binary.erl"},\n {line,\n 85}]},\n {mc_client_binary,\n get_dcp_docs_estimate,\n 3,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 754}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 616}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 225}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 226}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 430,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.16910.221>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 430}},\n infinity]}}}}'} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639668852989L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:34:12.989Z', u'text': u"Control connection to memcached on 'ns_1@172.23.110.64' disconnected. Check logs for details."} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639668852987L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:34:12.987Z', u'text': u"Service 'memcached' exited with status 137. Restarting. Messages:\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nW1216 06:57:58.114714 275614 HazptrDomain.h:670] Using the default inline executor for asynchronous reclamation may be susceptible to deadlock if the current thread happens to hold a resource needed by the deleter of a reclaimable object"} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666826735L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:00:26.735Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666826582L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:00:26.582Z', u'text': u'Started rebalancing bucket GleamBookUsers0'} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666826359L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:00:26.359Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.67','ns_1@172.23.110.64',\n 'ns_1@172.23.110.66','ns_1@172.23.110.65'], EjectNodes = ['ns_1@172.23.110.68'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 01d80168876166fe017db5c41d3a9b1b"} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666815787L, u'shortText': u'message', u'serverTime': u'2021-12-16T07:00:15.787Z', u'text': u'Bucket "GleamBookUsers0" loaded on node \'ns_1@172.23.110.64\' in 121 seconds.'} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673579L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.579Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.7383.221>,\n {{{{badmatch,{error,closed}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,151}]},\n {mc_client_binary,select_bucket,2,\n [{file,"src/mc_client_binary.erl"},\n {line,323}]},\n {ns_memcached_sockets_pool,take_socket,2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,82}]},\n {async,\'-async_init/4-fun-1-\',3,\n [{file,"src/async.erl"},{line,191}]}]},\n {gen_server,call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,454,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,<0.10010.216>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 454}},\n infinity]}}}}}.\nRebalance Operation Id = 5c2975a567250d185da25f2820c48c9d'} 2021-12-16 07:34:13,657 | test | ERROR | pool-3-thread-6 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639666673549L, u'shortText': u'message', u'serverTime': u'2021-12-16T06:57:53.549Z', u'text': u'Worker <0.7255.221> (for action {move,{454,\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.68\'],\n [\'ns_1@172.23.110.64\',\n \'ns_1@172.23.110.66\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.7383.221>,\n {{{{badmatch,\n {error,\n closed}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 151}]},\n {mc_client_binary,\n select_bucket,\n 2,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 323}]},\n {ns_memcached_sockets_pool,\n take_socket,\n 2,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 40}]},\n {ns_memcached_sockets_pool,\n \'-executing_on_socket/3-fun-0-\',\n 3,\n [{file,\n "src/ns_memcached_sockets_pool.erl"},\n {line,\n 82}]},\n {async,\n \'-async_init/4-fun-1-\',\n 3,\n [{file,\n "src/async.erl"},\n {line,\n 191}]}]},\n {gen_server,\n call,\n [\'ns_memcached-GleamBookUsers0\',\n {get_dcp_docs_estimate,\n 454,\n "replication:ns_1@172.23.110.64->ns_1@172.23.110.66:GleamBookUsers0"},\n 300000]}},\n {gen_server,\n call,\n [{\'janitor_agent-GleamBookUsers0\',\n \'ns_1@172.23.110.64\'},\n {if_rebalance,\n <0.10010.216>,\n {wait_dcp_data_move,\n [\'ns_1@172.23.110.66\'],\n 454}},\n infinity]}}}}'} 2021-12-16 07:34:13,658 | test | ERROR | pool-3-thread-6 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'4790835d4e7dbb6e01b4756829b0ead7', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=1c96663bcfa9be73e7f23a83601e8116', u'status': u'notRunning'} - rebalance failed 2021-12-16 07:35:11,698 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.67:8091 2021-12-16 07:35:11,700 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:35:11,713 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.67:8091 is running 2021-12-16 07:35:11,723 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.64:8091 2021-12-16 07:35:11,723 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:35:11,733 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:11,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:11,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:13,750 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:13,757 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:13,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:15,769 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:15,776 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:15,776 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:17,786 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:17,793 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:17,795 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:19,805 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:19,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:19,813 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:21,825 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:21,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:21,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:23,845 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:23,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:23,854 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:25,865 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:25,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:25,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:27,884 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:27,891 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:27,891 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:29,904 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:29,911 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:29,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:31,923 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:31,928 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:31,928 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:33,940 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:33,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:33,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:35,961 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:35,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:35,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:37,983 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:37,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:37,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:40,005 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:40,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:40,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:42,023 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:42,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:42,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:44,042 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:44,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:44,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:46,065 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:46,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:46,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:48,084 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:48,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:48,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:50,107 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:50,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:50,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:52,128 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:52,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:52,135 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:54,147 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:54,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:54,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:56,164 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:56,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:56,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:35:58,187 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:35:58,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:35:58,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:00,204 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:00,211 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:00,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:02,242 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:02,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:02,249 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:04,259 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:04,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:04,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:06,282 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:06,288 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:06,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:08,299 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:08,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:08,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:10,319 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:10,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:10,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:12,336 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:12,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:12,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:14,358 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:14,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:14,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:16,375 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:16,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:16,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:18,394 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:18,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:18,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:20,415 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:20,421 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:20,421 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:22,434 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:22,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:22,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:24,453 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:24,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:24,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:26,471 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:26,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:26,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:28,490 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:28,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:28,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:30,510 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:30,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:30,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:32,529 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:32,536 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:32,536 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:34,548 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:34,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:34,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:36,569 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:36,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:36,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:38,586 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:38,592 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:38,594 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:40,608 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:40,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:40,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:42,628 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:42,634 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:42,634 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:44,648 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:44,654 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:44,654 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:46,667 | test | WARNING | MainThread | [rest_client:is_ns_server_running:70] Server 172.23.110.64:8091 status is warmup 2021-12-16 07:36:46,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> is_ns_server_running():L82 2021-12-16 07:36:46,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-12-16 07:36:48,684 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.64:8091 is running 2021-12-16 07:36:48,694 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.65:8091 2021-12-16 07:36:48,694 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:36:48,703 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.65:8091 is running 2021-12-16 07:36:48,713 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:176] Waiting for ns_server @ 172.23.110.66:8091 2021-12-16 07:36:48,713 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:59] Checking if ns_server is running with timeout 1200secs 2021-12-16 07:36:48,723 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:179] ns_server @ 172.23.110.66:8091 is running 2021-12-16 07:36:48,724 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 07:36:48,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:36:49,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 07:36:49,230 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:36:49,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 07:36:49,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 07:36:49,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 07:36:49,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 07:36:49,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 07:36:49,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 07:36:49,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 07:36:49,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:36:49,395 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 07:36:49,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:36:49,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:36:49,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:36:49,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:49,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:49,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:49,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:49,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:49,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:49,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:49,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:49,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:49,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:49,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:49,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:49,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:36:49,990 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 07:36:49,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:36:50,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:50,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:36:50,026 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 07:36:50,026 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 07:36:50,029 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 07:36:50,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:36:50,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 07:36:50,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:36:50,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 07:36:50,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 07:36:50,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 07:36:50,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 07:36:50,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 07:36:50,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 07:36:50,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 07:36:50,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:36:50,569 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 07:36:50,594 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:36:50,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:50,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:50,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:50,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:50,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:50,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:50,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:50,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:50,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:50,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:50,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:50,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:51,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:51,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:51,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:51,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:51,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:51,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:51,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:51,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:51,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:51,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:51,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:51,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:51,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:51,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:51,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:51,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:51,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,780 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:51,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:51,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:51,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:51,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:51,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:51,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:52,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:52,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:36:52,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:36:52,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:36:52,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:36:52,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:36:52,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:36:52,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:36:52,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:36:52,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:36:52,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:36:52,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:52,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:36:52,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 07:36:52,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:36:52,592 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 07:36:52,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:36:52,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:52,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:36:52,634 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 07:36:52,634 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 07:36:52,638 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 07:36:52,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:36:53,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 07:36:53,124 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:36:53,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 07:36:53,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 07:36:53,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 07:36:53,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 07:36:53,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 07:36:53,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 07:36:53,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 07:36:53,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:36:53,286 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 07:36:53,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:36:53,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:53,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:53,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:53,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:53,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:53,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:53,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:53,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:53,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:53,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:53,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:53,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:53,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:53,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:53,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:53,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:53,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:54,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:54,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:54,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:54,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:54,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:54,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:54,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:54,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:54,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:54,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:54,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:54,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:54,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:54,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:54,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:54,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:36:54,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 07:36:54,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:36:54,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 07:36:54,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:54,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:54,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:36:54,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 07:36:55,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:55,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:55,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:36:55,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 07:36:55,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:55,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:55,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:36:55,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 07:36:55,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:36:55,165 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 07:36:55,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:36:55,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:55,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:36:55,198 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 07:36:55,198 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 07:36:55,201 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 07:36:55,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 07:36:55,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 07:36:55,667 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 07:36:55,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 07:36:55,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 07:36:55,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 07:36:55,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 07:36:55,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 07:36:55,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 07:36:55,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 07:36:55,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 07:36:55,770 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 07:36:55,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 07:36:55,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:55,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:55,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:36:55,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 07:36:55,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:55,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:56,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 07:36:56,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,127 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:56,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 07:36:56,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:56,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 07:36:56,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:56,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 07:36:56,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:56,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 07:36:56,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:56,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 07:36:56,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:56,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 07:36:56,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:56,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 07:36:56,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:56,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 07:36:56,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:56,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 07:36:56,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:56,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:56,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:56,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 07:36:57,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:57,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:57,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:57,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 07:36:57,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:57,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:57,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:57,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 07:36:57,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:57,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:57,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:57,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 07:36:57,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:57,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 07:36:57,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:57,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 07:36:57,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 07:36:57,424 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 07:36:57,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 07:36:57,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 07:36:57,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 07:36:57,480 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 07:36:57,480 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 07:36:57,480 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639666826.3 2021-12-16 07:36:57,480 | test | INFO | MainThread | [opd:abort_rebalance:807] Restarting Rebalance after killing at 60 2021-12-16 07:36:57,519 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639669017.48 2021-12-16 07:36:57,519 | infra | INFO | pool-3-thread-7 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639669017.48' started 2021-12-16 07:36:57,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> abort_rebalance():L812 2021-12-16 07:36:57,533 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 120 seconds. Reason: Let the rebalance begin after abort 2021-12-16 07:36:57,548 | test | DEBUG | pool-3-thread-7 | [task:start_rebalance:356] Removing node 172.23.110.68:8091 from cluster 2021-12-16 07:36:57,548 | test | DEBUG | pool-3-thread-7 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.68,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.68', 'user': 'Administrator'} 2021-12-16 07:36:57,605 | test | DEBUG | pool-3-thread-7 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 07:36:57,609 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+----------------+--------------+ | 172.23.110.67 | kv | 7.1.0-1913-enterprise | 4.65650448946 | Cluster node | | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 3.90015600624 | Cluster node | | 172.23.110.68 | kv | 7.1.0-1913-enterprise | 0.2209161769 | --- OUT ---> | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 7.33350066934 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 0.454349668617 | Cluster node | +---------------+----------+-----------------------+----------------+--------------+ 2021-12-16 07:36:57,621 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:36:57,621 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:36:57,641 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:36:57,644 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:36:57,644 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:07,657 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:07,657 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:07,677 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:07,680 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:07,680 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:17,694 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:17,696 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:17,713 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:17,714 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:17,714 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:27,727 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:27,727 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:27,746 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:27,747 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:27,749 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:37,760 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:37,760 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:37,779 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:37,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:37,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:47,790 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:47,792 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:47,808 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:47,811 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:47,811 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:37:57,821 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 07:37:57,822 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 07:37:57,835 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:37:57,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:37:57,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:07,849 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.67 % 2021-12-16 07:38:07,849 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 6.67 2021-12-16 07:38:07,871 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:07,874 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:07,874 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:17,884 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.33 % 2021-12-16 07:38:17,884 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.33 2021-12-16 07:38:17,900 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:17,903 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:17,903 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:27,914 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.33 % 2021-12-16 07:38:27,914 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.33 2021-12-16 07:38:27,937 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:27,940 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:27,940 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:37,950 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.33 % 2021-12-16 07:38:37,950 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.33 2021-12-16 07:38:37,967 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:37,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:37,971 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:47,987 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:38:47,987 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.8 2021-12-16 07:38:48,006 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:48,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:48,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:57,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:38:57,543 | test | INFO | MainThread | [opd:abort_rebalance:813] Rebalance % = 13.8 2021-12-16 07:38:57,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:38:57,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:38:57,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:38:58,023 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:38:58,023 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.8 2021-12-16 07:38:58,051 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:38:58,053 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:38:58,053 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:07,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:39:07,575 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:07,575 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:08,073 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:39:08,075 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.8 2021-12-16 07:39:08,095 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:08,102 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:08,102 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:17,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:39:17,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:17,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:18,117 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.80 % 2021-12-16 07:39:18,118 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 13.8 2021-12-16 07:39:18,141 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:18,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:18,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:27,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.46 % 2021-12-16 07:39:27,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:27,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:28,163 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.46 % 2021-12-16 07:39:28,164 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 20.46 2021-12-16 07:39:28,183 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:28,186 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:28,186 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:37,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.46 % 2021-12-16 07:39:37,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:37,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:38,197 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.46 % 2021-12-16 07:39:38,198 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 20.46 2021-12-16 07:39:38,220 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:38,223 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:38,223 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:47,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:39:47,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:47,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:48,236 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:39:48,236 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.13 2021-12-16 07:39:48,259 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:48,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:48,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:57,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:39:57,663 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:39:57,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:39:58,275 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:39:58,276 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.13 2021-12-16 07:39:58,292 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:39:58,296 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:39:58,296 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:07,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:40:07,694 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:07,694 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:08,308 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:40:08,308 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.13 2021-12-16 07:40:08,332 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:08,334 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:08,334 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:17,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:40:17,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:17,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:18,345 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 07:40:18,345 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.13 2021-12-16 07:40:18,362 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:18,367 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:18,367 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:27,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:27,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:27,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:28,380 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:28,380 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.59 2021-12-16 07:40:28,404 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:28,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:28,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:37,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:37,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:37,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:38,420 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:38,420 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.59 2021-12-16 07:40:38,443 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:38,446 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:38,447 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:47,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:47,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:47,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:48,459 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.59 % 2021-12-16 07:40:48,459 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 27.59 2021-12-16 07:40:48,479 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:48,483 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:48,483 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:57,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.26 % 2021-12-16 07:40:57,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:40:57,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:40:58,494 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:40:58,494 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:40:58,513 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:40:58,515 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:40:58,515 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:07,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:07,812 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:07,812 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:08,526 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:08,526 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:08,543 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:08,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:08,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:17,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:17,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:17,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:18,558 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:18,558 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:18,576 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:18,581 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:18,581 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:27,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:27,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:27,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:28,592 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:28,592 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:28,614 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:28,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:28,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:37,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:37,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:37,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:38,628 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:38,630 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:38,648 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:38,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:38,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:47,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:47,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:47,890 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:48,687 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:48,687 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:48,709 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:48,711 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:48,711 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:57,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:57,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:41:57,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:41:58,724 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:41:58,724 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:41:58,743 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:41:58,747 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:41:58,747 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:07,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:07,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:07,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:08,757 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:08,757 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:42:08,773 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:08,775 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:08,776 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:17,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:17,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:17,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:18,786 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:18,786 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:42:18,803 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:18,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:18,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:27,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:27,963 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:27,963 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:28,816 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:28,816 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:42:28,835 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:28,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:28,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:37,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:38,003 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:38,003 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:38,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:38,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:42:38,867 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:38,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:38,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:48,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:48,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:48,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:48,878 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.92 % 2021-12-16 07:42:48,880 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 40.92 2021-12-16 07:42:48,897 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:48,900 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:48,900 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:58,038 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.16 % 2021-12-16 07:42:58,045 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:42:58,045 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:42:58,911 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.16 % 2021-12-16 07:42:58,911 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.16 2021-12-16 07:42:58,934 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:42:58,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:42:58,936 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:08,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:08,072 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:08,072 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:08,947 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:08,947 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.39 2021-12-16 07:43:08,970 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:08,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:08,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:18,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:18,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:18,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:18,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:18,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.39 2021-12-16 07:43:19,002 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:19,005 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:19,005 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:28,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:28,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:28,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:29,016 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:29,016 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.39 2021-12-16 07:43:29,036 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:29,036 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:29,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:38,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:38,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:38,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:39,061 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:39,061 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.39 2021-12-16 07:43:39,088 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:39,091 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:39,091 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:48,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:48,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:48,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:49,105 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 07:43:49,105 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.39 2021-12-16 07:43:49,131 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:49,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:49,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:58,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.62 % 2021-12-16 07:43:58,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:43:58,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:43:59,145 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.62 % 2021-12-16 07:43:59,147 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.62 2021-12-16 07:43:59,167 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:43:59,171 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:43:59,171 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:08,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:08,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:08,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:09,187 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:09,187 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:09,207 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:09,209 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:09,209 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:18,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:18,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:18,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:19,224 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:19,226 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:19,250 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:19,255 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:19,255 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:28,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:28,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:28,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:29,266 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:29,266 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:29,285 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:29,288 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:29,288 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:38,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:38,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:38,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:39,299 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:39,299 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:39,334 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:39,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:39,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:48,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:48,308 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:48,308 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:49,351 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:49,352 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:49,371 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:49,374 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:49,374 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:58,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:58,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:44:58,332 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:44:59,395 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:44:59,397 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:44:59,421 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:44:59,424 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:44:59,424 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:08,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:08,348 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:08,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:09,440 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:09,440 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:45:09,466 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:09,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:09,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:18,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:18,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:18,365 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:19,480 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:19,480 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:45:19,505 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:19,506 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:19,506 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:28,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:28,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:28,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:29,519 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:29,519 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:45:29,533 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:29,536 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:29,536 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:38,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:38,403 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:38,403 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:39,555 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:39,556 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:45:39,579 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:39,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:39,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:48,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:48,423 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:48,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:49,598 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.85 % 2021-12-16 07:45:49,598 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 41.85 2021-12-16 07:45:49,621 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:49,622 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:49,624 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:58,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.08 % 2021-12-16 07:45:58,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:45:58,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:45:59,635 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.08 % 2021-12-16 07:45:59,635 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.08 2021-12-16 07:45:59,651 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:45:59,653 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:45:59,653 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:08,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:08,473 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:08,474 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:09,664 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:09,664 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.31 2021-12-16 07:46:09,684 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:09,687 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:09,687 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:18,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:18,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:18,496 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:19,698 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:19,698 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.31 2021-12-16 07:46:19,723 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:19,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:19,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:28,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:28,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:28,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:29,740 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:29,742 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.31 2021-12-16 07:46:29,769 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:29,772 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:29,773 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:38,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:38,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:38,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:39,783 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:39,783 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.31 2021-12-16 07:46:39,801 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:39,803 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:39,805 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:48,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:48,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:48,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:49,818 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.31 % 2021-12-16 07:46:49,819 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.31 2021-12-16 07:46:49,838 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:49,839 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:49,839 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:58,563 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.54 % 2021-12-16 07:46:58,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:46:58,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:46:59,855 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.54 % 2021-12-16 07:46:59,855 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.54 2021-12-16 07:46:59,871 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:46:59,875 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:46:59,875 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:08,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.54 % 2021-12-16 07:47:08,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:08,589 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:09,887 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.54 % 2021-12-16 07:47:09,888 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.54 2021-12-16 07:47:09,904 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:47:09,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:47:09,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:18,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:18,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:18,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:19,917 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:19,917 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:47:19,944 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:47:19,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:47:19,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:28,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:28,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:28,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:29,957 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:29,957 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:47:29,976 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:47:29,977 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:47:29,977 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:38,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:38,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:38,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:39,990 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:39,990 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:47:40,015 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:47:40,016 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:47:40,016 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:48,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:48,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:48,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:50,028 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:50,029 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:47:50,053 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:47:50,056 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:47:50,056 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:47:58,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:47:58,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:47:58,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:00,069 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:00,069 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:00,088 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:00,089 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:00,089 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:08,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:08,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:08,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:10,102 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:10,104 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:10,132 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:10,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:10,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:18,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:18,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:18,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:20,147 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:20,147 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:20,168 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:20,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:20,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:28,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:28,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:28,756 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:30,188 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:30,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:30,213 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:30,214 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:30,216 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:38,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:38,773 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:38,773 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:40,232 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:40,232 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:40,262 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:40,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:40,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:48,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:48,796 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:48,796 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:50,280 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:50,282 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:48:50,301 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:48:50,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:48:50,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:48:58,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:48:58,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:48:58,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:00,321 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.77 % 2021-12-16 07:49:00,321 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 42.77 2021-12-16 07:49:00,348 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:00,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:00,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:08,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 07:49:08,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:08,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:10,362 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.01 % 2021-12-16 07:49:10,362 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.01 2021-12-16 07:49:10,380 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:10,382 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:10,382 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:18,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:18,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:18,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:20,395 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:20,395 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.24 2021-12-16 07:49:20,420 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:20,421 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:20,421 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:28,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:28,880 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:28,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:30,433 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:30,434 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.24 2021-12-16 07:49:30,451 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:30,454 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:30,454 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:38,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:38,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:38,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:40,469 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:40,469 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.24 2021-12-16 07:49:40,493 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:40,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:40,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:48,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:48,918 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:48,918 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:50,507 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:50,507 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.24 2021-12-16 07:49:50,525 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:49:50,528 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:49:50,528 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:49:58,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.24 % 2021-12-16 07:49:58,940 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:49:58,940 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:00,539 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.47 % 2021-12-16 07:50:00,539 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.47 2021-12-16 07:50:00,556 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:00,559 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:00,559 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:08,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.47 % 2021-12-16 07:50:08,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:08,961 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:10,572 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.47 % 2021-12-16 07:50:10,572 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.47 2021-12-16 07:50:10,595 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:10,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:10,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:18,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:18,977 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:18,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:20,608 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:20,608 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:50:20,625 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:20,625 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:20,627 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:28,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:29,009 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:29,009 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:30,638 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:30,638 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:50:30,658 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:30,661 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:30,661 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:39,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:39,026 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:39,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:40,673 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:40,673 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:50:40,690 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:40,694 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:40,694 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:49,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:49,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:49,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:50,709 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:50,710 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:50:50,736 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:50:50,740 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:50:50,740 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:50:59,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:50:59,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:50:59,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:00,755 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:00,755 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:00,780 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:00,782 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:00,782 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:09,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:09,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:09,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:10,795 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:10,796 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:10,815 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:10,819 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:10,819 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:19,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:19,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:19,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:20,832 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:20,832 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:20,846 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:20,848 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:20,848 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:29,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:29,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:29,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:30,859 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:30,861 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:30,880 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:30,882 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:30,882 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:39,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:39,151 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:39,151 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:40,897 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:40,897 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:40,917 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:40,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:40,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:49,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:49,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:49,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:50,931 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:50,933 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:51:50,956 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:51:50,957 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:51:50,957 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:51:59,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:51:59,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:51:59,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:00,970 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:52:00,970 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.7 2021-12-16 07:52:00,996 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:01,000 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:01,000 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:09,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.70 % 2021-12-16 07:52:09,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:09,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:11,012 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.93 % 2021-12-16 07:52:11,012 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.93 2021-12-16 07:52:11,029 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:11,033 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:11,033 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:19,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.93 % 2021-12-16 07:52:19,234 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:19,234 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:21,046 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.93 % 2021-12-16 07:52:21,046 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 43.93 2021-12-16 07:52:21,078 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:21,081 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:21,081 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:29,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:29,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:29,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:31,098 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:31,098 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.16 2021-12-16 07:52:31,121 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:31,124 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:31,124 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:39,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:39,270 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:39,270 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:41,138 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:41,138 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.16 2021-12-16 07:52:41,155 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:41,157 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:41,157 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:49,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:49,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:49,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:51,170 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:51,170 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.16 2021-12-16 07:52:51,187 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:52:51,190 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:52:51,190 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:52:59,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:52:59,316 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:52:59,316 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:01,203 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.16 % 2021-12-16 07:53:01,203 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.16 2021-12-16 07:53:01,220 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:01,223 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:01,223 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:09,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.39 % 2021-12-16 07:53:09,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:09,335 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:11,236 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.39 % 2021-12-16 07:53:11,236 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.39 2021-12-16 07:53:11,255 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:11,257 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:11,257 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:19,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:19,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:19,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:21,270 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:21,270 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:53:21,296 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:21,299 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:21,301 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:29,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:29,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:29,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:31,313 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:31,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:53:31,336 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:31,341 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:31,341 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:39,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:39,398 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:39,398 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:41,361 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:41,361 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:53:41,394 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:41,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:41,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:49,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:49,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:49,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:51,411 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:51,414 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:53:51,430 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:53:51,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:53:51,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:53:59,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:53:59,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:53:59,440 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:01,446 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:01,446 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:01,473 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:01,476 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:01,476 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:09,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:09,457 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:09,457 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:11,490 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:11,490 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:11,513 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:11,515 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:11,515 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:19,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:19,476 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:19,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:21,526 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:21,526 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:21,545 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:21,548 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:21,549 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:29,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:29,493 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:29,493 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:31,559 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:31,559 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:31,576 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:31,578 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:31,578 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:39,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:39,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:39,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:41,591 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:41,591 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:41,617 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:41,618 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:41,619 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:49,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:49,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:49,529 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:51,634 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:51,634 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:54:51,654 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:54:51,657 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:54:51,657 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:54:59,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:54:59,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:54:59,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:01,671 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:55:01,671 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:55:01,690 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:01,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:01,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:09,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:55:09,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:09,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:11,710 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.62 % 2021-12-16 07:55:11,710 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.62 2021-12-16 07:55:11,733 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:11,736 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:11,736 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:19,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.86 % 2021-12-16 07:55:19,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:19,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:21,749 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.86 % 2021-12-16 07:55:21,749 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 44.86 2021-12-16 07:55:21,766 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:21,769 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:21,769 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:29,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.86 % 2021-12-16 07:55:29,637 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:29,637 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:31,782 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:31,782 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.09 2021-12-16 07:55:31,805 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:31,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:31,808 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:39,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:39,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:39,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:41,822 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:41,822 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.09 2021-12-16 07:55:41,838 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:41,842 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:41,842 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:49,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:49,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:49,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:51,852 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:51,852 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.09 2021-12-16 07:55:51,868 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:55:51,869 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:55:51,869 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:55:59,709 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:55:59,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:55:59,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:01,881 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:56:01,881 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.09 2021-12-16 07:56:01,898 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:01,900 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:01,900 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:09,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.09 % 2021-12-16 07:56:09,749 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:09,749 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:11,911 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.32 % 2021-12-16 07:56:11,911 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.32 2021-12-16 07:56:11,930 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:11,933 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:11,933 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:19,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.32 % 2021-12-16 07:56:19,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:19,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:21,944 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.32 % 2021-12-16 07:56:21,944 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.32 2021-12-16 07:56:21,969 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:21,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:21,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:29,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:29,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:29,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:31,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:31,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:56:32,003 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:32,006 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:32,006 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:39,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:39,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:39,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:42,017 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:42,017 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:56:42,035 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:42,036 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:42,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:49,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:49,831 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:49,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:52,049 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:52,049 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:56:52,075 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:56:52,076 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:56:52,078 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:56:59,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:56:59,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:56:59,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:02,088 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:02,089 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:02,118 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:02,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:02,121 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:09,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:09,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:57:09,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:12,134 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:12,134 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:12,160 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:12,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:12,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:19,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:19,907 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:57:19,908 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:22,174 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:22,174 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:22,200 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:22,206 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:22,206 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:29,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:29,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:57:29,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:32,219 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:32,219 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:32,239 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:32,242 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:32,242 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:39,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:39,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:57:39,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:42,255 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:42,255 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:42,279 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:42,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:42,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:49,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:49,982 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:57:49,982 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:52,295 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:57:52,295 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:57:52,318 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:57:52,321 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:57:52,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:57:59,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:00,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:00,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:02,334 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:02,335 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:58:02,351 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:02,355 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:02,355 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:10,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:10,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:10,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:12,375 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:12,377 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:58:12,407 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:12,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:12,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:20,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:20,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:20,043 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:22,430 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.55 % 2021-12-16 07:58:22,430 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.55 2021-12-16 07:58:22,453 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:22,456 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:22,456 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:30,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.78 % 2021-12-16 07:58:30,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:30,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:32,470 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.78 % 2021-12-16 07:58:32,470 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 45.78 2021-12-16 07:58:32,493 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:32,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:32,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:40,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:58:40,092 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:40,092 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:42,512 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:58:42,512 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.01 2021-12-16 07:58:42,533 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:42,539 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:42,539 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:50,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:58:50,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:58:50,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:58:52,552 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:58:52,552 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.01 2021-12-16 07:58:52,575 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:58:52,575 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:58:52,576 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:00,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:59:00,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:00,128 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:02,588 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:59:02,589 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.01 2021-12-16 07:59:02,611 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:02,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:02,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:10,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:59:10,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:10,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:12,625 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.01 % 2021-12-16 07:59:12,625 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.01 2021-12-16 07:59:12,641 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:12,644 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:12,644 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:20,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.24 % 2021-12-16 07:59:20,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:20,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:22,655 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.24 % 2021-12-16 07:59:22,655 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.24 2021-12-16 07:59:22,674 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:22,676 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:22,676 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:30,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.24 % 2021-12-16 07:59:30,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:30,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:32,690 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 07:59:32,690 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 07:59:32,710 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:32,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:32,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:40,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 07:59:40,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:40,211 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:42,726 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 07:59:42,726 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 07:59:42,746 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:42,749 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:42,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:50,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 07:59:50,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 07:59:50,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 07:59:52,762 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 07:59:52,762 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 07:59:52,786 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 07:59:52,788 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 07:59:52,788 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:00,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:00,249 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:00,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:02,805 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:02,805 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:02,832 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:02,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:02,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:10,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:10,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:10,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:12,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:12,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:12,867 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:12,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:12,869 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:20,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:20,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:20,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:22,881 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:22,881 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:22,905 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:22,907 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:22,907 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:30,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:30,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:30,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:32,921 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:32,921 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:32,947 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:32,950 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:32,951 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:40,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:40,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:40,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:42,963 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:42,963 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:42,980 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:42,983 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:42,984 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:50,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:50,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:00:50,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:00:53,000 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:00:53,000 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:00:53,025 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:00:53,026 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:00:53,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:00,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:00,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:00,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:03,039 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:03,040 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:01:03,062 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:03,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:03,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:10,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:10,394 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:10,394 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:13,078 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:13,078 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:01:13,098 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:13,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:13,102 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:20,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:20,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:20,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:23,115 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 08:01:23,115 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.47 2021-12-16 08:01:23,132 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:23,135 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:23,135 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:30,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.71 % 2021-12-16 08:01:30,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:30,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:33,150 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.71 % 2021-12-16 08:01:33,151 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.71 2021-12-16 08:01:33,174 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:33,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:33,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:40,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:01:40,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:40,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:43,187 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:01:43,187 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.94 2021-12-16 08:01:43,211 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:43,214 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:43,214 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:50,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:01:50,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:01:50,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:01:53,226 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:01:53,226 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.94 2021-12-16 08:01:53,246 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:01:53,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:01:53,249 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:00,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:02:00,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:00,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:03,263 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:02:03,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.94 2021-12-16 08:02:03,288 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:03,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:03,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:10,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:02:10,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:10,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:13,303 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.94 % 2021-12-16 08:02:13,303 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 46.94 2021-12-16 08:02:13,342 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:13,346 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:13,346 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:20,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.17 % 2021-12-16 08:02:20,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:20,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:23,361 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.17 % 2021-12-16 08:02:23,361 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.17 2021-12-16 08:02:23,388 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:23,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:23,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:30,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.17 % 2021-12-16 08:02:30,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:30,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:33,404 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:02:33,404 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:02:33,424 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:33,426 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:33,426 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:40,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:02:40,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:40,579 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:43,438 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:02:43,438 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:02:43,463 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:43,464 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:43,464 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:50,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:02:50,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:02:50,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:02:53,480 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:02:53,482 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:02:53,500 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:02:53,503 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:02:53,503 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:00,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:00,618 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:00,618 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:03,526 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:03,528 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:03,546 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:03,548 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:03,548 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:10,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:10,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:10,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:13,561 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:13,561 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:13,595 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:13,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:13,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:20,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:20,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:20,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:23,614 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:23,614 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:23,650 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:23,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:23,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:30,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:30,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:30,676 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:33,665 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:33,665 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:33,690 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:33,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:33,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:40,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:40,704 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:40,704 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:43,703 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:43,703 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:43,727 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:43,730 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:43,730 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:50,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:50,720 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:03:50,720 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:03:53,743 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:03:53,743 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:03:53,760 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:03:53,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:03:53,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:00,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:00,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:00,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:03,776 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:03,776 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:04:03,801 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:03,802 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:03,802 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:10,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:10,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:10,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:13,815 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:13,815 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:04:13,834 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:13,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:13,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:20,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:20,779 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:20,779 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:23,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:23,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:04:23,864 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:23,865 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:23,867 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:30,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:30,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:30,802 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:33,878 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.40 % 2021-12-16 08:04:33,878 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.4 2021-12-16 08:04:33,898 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:33,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:33,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:40,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.63 % 2021-12-16 08:04:40,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:40,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:43,911 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.63 % 2021-12-16 08:04:43,911 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.63 2021-12-16 08:04:43,928 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:43,931 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:43,933 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:50,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:04:50,854 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:04:50,854 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:04:53,944 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:04:53,944 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.86 2021-12-16 08:04:53,960 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:04:53,961 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:04:53,961 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:00,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:00,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:00,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:03,973 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:03,973 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.86 2021-12-16 08:05:03,990 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:03,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:03,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:10,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:10,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:10,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:14,010 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:14,010 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.86 2021-12-16 08:05:14,033 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:14,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:14,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:20,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:20,905 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:20,907 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:24,049 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.86 % 2021-12-16 08:05:24,051 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 47.86 2021-12-16 08:05:24,066 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:24,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:24,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:30,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.09 % 2021-12-16 08:05:30,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:30,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:34,082 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.09 % 2021-12-16 08:05:34,082 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.09 2021-12-16 08:05:34,099 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:34,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:34,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:40,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:05:40,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:40,944 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:44,112 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:05:44,112 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:05:44,131 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:44,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:44,134 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:50,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:05:50,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:05:50,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:05:54,145 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:05:54,145 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:05:54,164 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:05:54,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:05:54,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:00,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:00,979 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:00,979 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:04,177 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:04,177 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:04,201 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:04,203 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:04,203 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:10,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:11,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:11,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:14,216 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:14,216 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:14,233 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:14,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:14,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:21,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:21,025 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:21,026 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:24,246 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:24,246 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:24,265 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:24,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:24,267 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:31,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:31,051 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:31,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:34,279 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:34,279 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:34,301 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:34,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:34,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:41,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:41,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:41,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:44,313 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:44,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:44,329 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:44,332 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:44,332 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:51,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:51,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:06:51,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:06:54,342 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:06:54,342 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:06:54,359 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:06:54,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:06:54,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:01,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:01,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:01,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:04,374 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:04,374 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:07:04,391 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:04,394 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:04,394 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:11,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:11,121 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:11,121 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:14,407 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:14,407 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:07:14,430 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:14,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:14,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:21,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:21,138 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:21,138 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:24,447 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:24,447 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:07:24,463 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:24,466 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:24,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:31,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:31,160 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:31,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:34,479 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:34,479 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:07:34,496 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:34,497 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:34,499 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:41,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:41,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:41,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:44,509 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:44,509 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.32 2021-12-16 08:07:44,525 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:44,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:44,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:51,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.32 % 2021-12-16 08:07:51,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:07:51,201 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:07:54,540 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.55 % 2021-12-16 08:07:54,542 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.55 2021-12-16 08:07:54,566 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:07:54,568 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:07:54,569 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:01,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:01,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:01,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:04,585 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:04,585 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-16 08:08:04,608 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:04,611 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:04,611 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:11,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:11,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:11,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:14,621 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:14,621 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-16 08:08:14,638 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:14,640 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:14,640 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:21,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:21,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:21,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:24,651 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:24,653 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-16 08:08:24,677 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:24,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:24,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:31,280 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:31,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:31,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:34,691 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:34,691 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-16 08:08:34,713 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:34,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:34,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:41,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:41,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:41,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:44,727 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.79 % 2021-12-16 08:08:44,729 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 48.79 2021-12-16 08:08:44,743 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:44,746 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:44,746 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:51,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.02 % 2021-12-16 08:08:51,341 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:08:51,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:08:54,759 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:08:54,759 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:08:54,782 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:08:54,783 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:08:54,783 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:01,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:01,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:01,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:04,796 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:04,796 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:04,813 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:04,816 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:04,816 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:11,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:11,381 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:11,381 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:14,828 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:14,828 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:14,851 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:14,854 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:14,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:21,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:21,397 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:21,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:24,867 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:24,867 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:24,888 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:24,891 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:24,891 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:31,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:31,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:31,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:34,901 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:34,903 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:34,920 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:34,923 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:34,923 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:41,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:41,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:41,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:44,934 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:44,934 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:44,951 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:44,953 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:44,953 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:51,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:51,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:09:51,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:09:54,967 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:09:54,969 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:09:54,994 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:09:54,997 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:09:54,997 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:01,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:01,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:01,471 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:05,009 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:05,009 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:05,030 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:05,033 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:05,033 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:11,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:11,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:11,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:15,045 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:15,045 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:15,063 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:15,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:15,066 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:21,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:21,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:21,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:25,079 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:25,081 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:25,115 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:25,118 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:25,118 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:31,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:31,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:31,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:35,131 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:35,131 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:35,154 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:35,157 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:35,157 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:41,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:41,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:41,542 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:45,168 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:45,170 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:45,196 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:45,198 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:45,198 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:51,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:51,559 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:10:51,559 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:10:55,210 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.25 % 2021-12-16 08:10:55,210 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.25 2021-12-16 08:10:55,233 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:10:55,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:10:55,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:01,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.48 % 2021-12-16 08:11:01,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:01,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:05,250 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:05,252 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.71 2021-12-16 08:11:05,280 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:05,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:05,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:11,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:11,596 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:11,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:15,296 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:15,298 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.71 2021-12-16 08:11:15,325 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:15,329 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:15,331 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:21,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:21,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:21,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:25,345 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:25,345 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.71 2021-12-16 08:11:25,371 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:25,375 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:25,375 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:31,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:31,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:31,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:35,387 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:35,387 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.71 2021-12-16 08:11:35,408 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:35,413 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:35,413 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:41,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:41,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:41,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:45,424 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.71 % 2021-12-16 08:11:45,424 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 49.71 2021-12-16 08:11:45,440 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:45,443 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:45,444 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:51,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.94 % 2021-12-16 08:11:51,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:11:51,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:11:55,456 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:11:55,456 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:11:55,471 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:11:55,473 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:11:55,473 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:01,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:01,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:01,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:05,506 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:05,506 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:05,528 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:05,532 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:05,532 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:11,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:11,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:11,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:15,542 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:15,543 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:15,569 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:15,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:15,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:21,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:21,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:21,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:25,588 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:25,588 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:25,608 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:25,609 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:25,609 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:31,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:31,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:31,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:35,622 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:35,624 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:35,648 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:35,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:35,655 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:41,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:41,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:41,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:45,670 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:45,670 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:45,687 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:45,688 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:45,688 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:51,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:51,783 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:12:51,783 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:12:55,700 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:12:55,701 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:12:55,724 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:12:55,726 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:12:55,726 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:01,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:01,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:01,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:05,740 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:05,740 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:05,757 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:05,760 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:05,760 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:11,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:11,818 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:11,818 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:15,773 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:15,773 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:15,792 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:15,795 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:15,795 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:21,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:21,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:21,838 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:25,809 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:25,809 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:25,834 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:25,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:25,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:31,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:31,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:31,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:35,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:35,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:35,867 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:35,869 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:35,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:41,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:41,871 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:41,871 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:45,882 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:45,882 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:45,900 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:45,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:45,903 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:51,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:51,897 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:13:51,897 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:13:55,914 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:13:55,914 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.17 2021-12-16 08:13:55,938 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:13:55,940 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:13:55,940 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:01,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.17 % 2021-12-16 08:14:01,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:01,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:05,951 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.40 % 2021-12-16 08:14:05,951 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.4 2021-12-16 08:14:05,970 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:05,974 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:05,974 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:11,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:11,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:11,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:15,986 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:15,987 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.64 2021-12-16 08:14:16,013 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:16,015 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:16,016 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:21,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:21,954 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:21,954 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:26,028 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:26,028 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.64 2021-12-16 08:14:26,066 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:26,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:26,071 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:31,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:31,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:31,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:36,082 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:36,082 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.64 2021-12-16 08:14:36,098 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:36,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:36,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:41,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:41,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:41,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:46,112 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:46,114 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.64 2021-12-16 08:14:46,138 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:46,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:46,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:52,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.64 % 2021-12-16 08:14:52,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:14:52,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:14:56,154 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.87 % 2021-12-16 08:14:56,154 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 50.87 2021-12-16 08:14:56,180 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:14:56,183 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:14:56,183 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:02,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:02,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:02,035 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:06,194 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:06,194 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:06,210 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:06,211 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:06,211 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:12,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:12,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:12,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:16,223 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:16,223 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:16,237 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:16,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:16,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:22,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:22,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:22,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:26,250 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:26,252 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:26,269 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:26,273 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:26,273 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:32,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:32,105 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:32,105 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:36,283 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:36,283 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:36,301 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:36,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:36,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:42,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:42,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:42,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:46,315 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:46,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:46,341 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:46,342 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:46,342 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:52,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:52,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:15:52,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:15:56,355 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:15:56,355 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:15:56,377 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:15:56,378 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:15:56,378 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:02,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:02,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:02,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:06,388 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:06,390 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:06,405 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:06,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:06,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:12,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:12,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:12,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:16,417 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:16,418 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:16,437 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:16,440 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:16,440 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:22,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:22,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:22,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:26,450 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:26,451 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:26,469 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:26,470 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:26,470 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:32,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:32,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:32,223 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:36,483 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:36,483 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:36,509 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:36,512 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:36,512 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:42,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:42,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:42,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:46,523 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:46,523 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:46,542 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:46,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:46,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:52,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:52,265 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:16:52,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:16:56,556 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:16:56,556 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:16:56,572 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:16:56,575 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:16:56,575 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:02,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:17:02,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:02,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:06,588 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.10 % 2021-12-16 08:17:06,588 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.1 2021-12-16 08:17:06,614 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:06,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:06,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:12,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.33 % 2021-12-16 08:17:12,298 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:12,298 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:16,630 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:16,630 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:17:16,647 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:16,650 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:16,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:22,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:22,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:22,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:26,663 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:26,663 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:17:26,687 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:26,688 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:26,688 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:32,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:32,334 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:32,334 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:36,698 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:36,698 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:17:36,717 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:36,720 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:36,720 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:42,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:42,357 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:42,357 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:46,732 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:46,733 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:17:46,750 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:46,753 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:46,753 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:52,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:52,387 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:17:52,387 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:17:56,765 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:17:56,766 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:17:56,785 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:17:56,786 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:17:56,786 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:02,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:18:02,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:02,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:06,798 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.56 % 2021-12-16 08:18:06,798 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 51.56 2021-12-16 08:18:06,815 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:06,818 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:06,818 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:12,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:12,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:12,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:16,828 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:16,829 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:18:16,854 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:16,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:16,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:22,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:22,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:22,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:26,875 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:26,875 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:18:26,898 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:26,903 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:26,904 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:32,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:32,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:32,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:36,914 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:36,914 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:18:36,931 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:36,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:36,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:42,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:42,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:42,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:46,947 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:46,947 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:18:46,971 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:46,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:46,974 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:52,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:52,513 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:18:52,513 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:18:56,986 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:18:56,986 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:18:57,006 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:18:57,012 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:18:57,012 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:02,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:02,533 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:02,533 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:07,023 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:07,023 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:07,043 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:07,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:07,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:12,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:12,552 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:12,552 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:17,061 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:17,062 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:17,084 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:17,086 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:17,086 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:22,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:22,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:22,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:27,098 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:27,098 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:27,118 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:27,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:27,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:32,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:32,591 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:32,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:37,131 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:37,131 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:37,157 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:37,158 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:37,158 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:42,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:42,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:42,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:47,171 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:47,173 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:47,196 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:47,200 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:47,201 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:52,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:52,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:19:52,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:19:57,213 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:19:57,213 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:19:57,237 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:19:57,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:19:57,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:02,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:20:02,653 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:02,653 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:07,252 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:20:07,253 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.02 2021-12-16 08:20:07,278 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:07,279 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:07,279 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:12,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.02 % 2021-12-16 08:20:12,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:12,673 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:17,298 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.25 % 2021-12-16 08:20:17,299 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.25 2021-12-16 08:20:17,319 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:17,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:17,325 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:22,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:22,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:22,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:27,335 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:27,335 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.49 2021-12-16 08:20:27,351 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:27,354 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:27,355 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:32,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:32,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:32,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:37,367 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:37,367 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.49 2021-12-16 08:20:37,391 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:37,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:37,392 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:42,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:42,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:42,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:47,404 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:47,404 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.49 2021-12-16 08:20:47,424 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:47,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:47,428 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:52,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:52,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:20:52,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:20:57,440 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:20:57,440 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.49 2021-12-16 08:20:57,464 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:20:57,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:20:57,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:02,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:21:02,769 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:02,769 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:07,479 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:21:07,480 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.49 2021-12-16 08:21:07,500 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:07,503 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:07,503 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:12,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.49 % 2021-12-16 08:21:12,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:12,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:17,515 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:17,515 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:21:17,539 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:17,540 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:17,542 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:22,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:22,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:22,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:27,555 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:27,555 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:21:27,582 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:27,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:27,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:32,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:32,822 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:32,822 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:37,596 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:37,596 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:21:37,615 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:37,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:37,618 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:42,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:42,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:42,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:47,628 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:47,628 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:21:47,645 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:47,648 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:47,648 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:52,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:52,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:21:52,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:21:57,667 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:21:57,667 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:21:57,688 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:21:57,691 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:21:57,691 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:02,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:02,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:02,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:07,704 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:07,704 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:07,721 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:07,723 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:07,723 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:12,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:12,910 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:12,911 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:17,734 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:17,736 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:17,759 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:17,762 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:17,762 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:22,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:22,926 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:22,926 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:27,776 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:27,776 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:27,795 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:27,799 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:27,799 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:32,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:32,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:32,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:37,812 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:37,812 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:37,835 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:37,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:37,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:42,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:42,993 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:42,993 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:47,849 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:47,849 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:47,868 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:47,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:47,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:53,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:53,012 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:22:53,013 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:22:57,882 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:22:57,884 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:22:57,901 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:22:57,904 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:22:57,904 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:03,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:23:03,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:03,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:07,915 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:23:07,915 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:23:07,940 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:07,943 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:07,943 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:13,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:23:13,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:13,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:17,970 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.95 % 2021-12-16 08:23:17,970 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 52.95 2021-12-16 08:23:17,990 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:17,994 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:17,994 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:23,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:23,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:23,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:28,003 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:28,005 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.41 2021-12-16 08:23:28,022 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:28,025 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:28,025 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:33,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:33,094 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:33,094 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:38,038 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:38,039 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.41 2021-12-16 08:23:38,061 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:38,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:38,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:43,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:43,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:43,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:48,079 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:48,079 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.41 2021-12-16 08:23:48,105 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:48,108 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:48,108 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:53,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:53,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:23:53,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:23:58,118 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:23:58,118 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.41 2021-12-16 08:23:58,142 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:23:58,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:23:58,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:03,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:24:03,155 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:03,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:08,155 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:24:08,155 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.41 2021-12-16 08:24:08,180 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:08,181 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:08,181 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:13,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.41 % 2021-12-16 08:24:13,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:13,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:18,209 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:18,209 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:24:18,224 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:18,229 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:18,229 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:23,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:23,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:23,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:28,240 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:28,242 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:24:28,260 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:28,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:28,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:33,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:33,207 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:33,207 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:38,276 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:38,276 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:24:38,302 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:38,305 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:38,305 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:43,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:43,230 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:43,230 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:48,316 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:48,316 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:24:48,334 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:48,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:48,336 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:53,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:53,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:24:53,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:24:58,351 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:24:58,351 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:24:58,375 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:24:58,380 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:24:58,380 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:03,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:03,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:03,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:08,391 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:08,391 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:08,407 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:08,408 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:08,408 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:13,280 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:13,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:13,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:18,423 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:18,424 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:18,443 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:18,447 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:18,447 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:23,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:23,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:23,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:28,460 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:28,460 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:28,486 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:28,490 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:28,492 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:33,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:33,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:33,346 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:38,502 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:38,503 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:38,519 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:38,522 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:38,522 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:43,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:43,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:43,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:48,539 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:48,539 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:48,562 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:48,565 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:48,566 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:53,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:53,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:25:53,397 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:25:58,578 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:25:58,579 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:25:58,605 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:25:58,608 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:25:58,608 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:03,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:26:03,413 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:03,413 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:08,619 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:26:08,619 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:26:08,642 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:08,645 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:08,645 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:13,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:26:13,427 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:13,427 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:18,661 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:26:18,663 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 53.87 2021-12-16 08:26:18,680 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:18,684 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:18,684 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:23,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.87 % 2021-12-16 08:26:23,441 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:23,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:28,696 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:28,696 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.34 2021-12-16 08:26:28,721 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:28,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:28,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:33,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:33,461 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:33,461 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:38,734 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:38,736 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.34 2021-12-16 08:26:38,753 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:38,756 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:38,756 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:43,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:43,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:43,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:48,770 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:48,770 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.34 2021-12-16 08:26:48,792 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:48,796 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:48,796 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:53,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.34 % 2021-12-16 08:26:53,500 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:26:53,500 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:26:58,808 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:26:58,808 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:26:58,826 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:26:58,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:26:58,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:03,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:03,523 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:03,523 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:08,842 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:08,842 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:08,865 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:08,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:08,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:13,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:13,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:13,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:18,878 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:18,878 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:18,894 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:18,895 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:18,897 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:23,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:23,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:23,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:28,907 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:28,907 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:28,933 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:28,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:28,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:33,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:33,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:33,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:38,947 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:38,947 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:38,966 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:38,969 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:38,969 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:43,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:43,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:43,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:48,980 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:48,980 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:48,996 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:48,997 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:48,997 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:53,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:53,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:27:53,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:27:59,010 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:27:59,012 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:27:59,036 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:27:59,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:27:59,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:03,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:03,642 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:03,642 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:09,048 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:09,049 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:09,065 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:09,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:09,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:13,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:13,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:13,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:19,079 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:19,081 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:19,095 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:19,098 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:19,098 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:23,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:23,677 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:23,678 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:29,111 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:29,111 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:29,141 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:29,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:29,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:33,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:33,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:33,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:39,155 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:39,155 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:39,173 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:39,174 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:39,174 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:43,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:43,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:43,711 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:49,187 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:49,187 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:49,207 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:49,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:49,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:53,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:53,744 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:28:53,744 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:28:59,224 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:28:59,224 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:28:59,247 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:28:59,252 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:28:59,252 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:03,755 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:03,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:03,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:09,262 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:09,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:09,278 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:09,279 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:09,279 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:13,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:13,780 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:13,780 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:19,290 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:19,292 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:19,315 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:19,316 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:19,316 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:23,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:23,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:23,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:29,335 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:29,335 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:29,361 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:29,367 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:29,368 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:33,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:33,825 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:33,826 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:39,378 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:39,378 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:39,404 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:39,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:39,407 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:43,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:43,844 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:43,844 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:49,420 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:49,420 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:49,440 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:49,444 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:49,444 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:53,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:53,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:29:53,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:29:59,454 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:29:59,456 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 54.8 2021-12-16 08:29:59,474 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:29:59,476 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:29:59,477 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:03,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.80 % 2021-12-16 08:30:03,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:03,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:09,492 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.49 % 2021-12-16 08:30:09,492 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.49 2021-12-16 08:30:09,519 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:09,523 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:09,523 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:13,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:13,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:13,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:19,535 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:19,535 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:30:19,555 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:19,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:19,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:23,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:23,924 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:23,924 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:29,575 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:29,576 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:30:29,605 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:29,608 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:29,608 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:33,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:33,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:33,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:39,622 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:39,622 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:30:39,648 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:39,650 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:39,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:43,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:43,960 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:43,960 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:49,663 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:49,663 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:30:49,687 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:49,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:49,693 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:53,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:53,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:30:53,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:30:59,703 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:30:59,703 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:30:59,720 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:30:59,721 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:30:59,721 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:03,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:04,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:04,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:09,733 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:09,733 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:09,760 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:09,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:09,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:14,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:14,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:14,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:19,778 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:19,779 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:19,799 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:19,802 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:19,802 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:24,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:24,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:24,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:29,813 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:29,813 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:29,832 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:29,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:29,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:34,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:34,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:34,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:39,846 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:39,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:39,871 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:39,872 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:39,872 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:44,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:44,078 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:44,078 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:49,884 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:49,884 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:49,904 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:49,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:49,905 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:54,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:54,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:31:54,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:31:59,918 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:31:59,918 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:31:59,943 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:31:59,946 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:31:59,946 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:04,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:04,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:04,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:09,954 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:09,956 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:32:09,974 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:32:09,977 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:32:09,977 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:14,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:14,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:14,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:19,990 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:19,992 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:32:20,016 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:32:20,019 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:32:20,019 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:24,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:24,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:24,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:30,030 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:30,030 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:32:30,049 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:32:30,053 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:32:30,053 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:34,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:34,184 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:34,184 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:40,062 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:40,063 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:32:40,079 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:32:40,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:32:40,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:44,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:44,209 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:44,209 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:50,092 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:50,092 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:32:50,112 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:32:50,114 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:32:50,115 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:32:54,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:32:54,226 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:32:54,226 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:00,124 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:33:00,125 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 55.72 2021-12-16 08:33:00,141 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:00,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:00,144 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:04,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.72 % 2021-12-16 08:33:04,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:04,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:10,157 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.42 % 2021-12-16 08:33:10,157 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.42 2021-12-16 08:33:10,174 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:10,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:10,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:14,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:14,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:14,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:20,190 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:20,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:33:20,209 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:20,211 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:20,211 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:24,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:24,279 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:24,279 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:30,223 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:30,224 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:33:30,246 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:30,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:30,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:34,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:34,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:34,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:40,259 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:40,259 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:33:40,275 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:40,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:40,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:44,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:44,319 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:44,319 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:50,286 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:50,286 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:33:50,303 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:33:50,305 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:33:50,305 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:33:54,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:33:54,338 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:33:54,338 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:00,315 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:00,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:00,332 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:00,334 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:00,334 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:04,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:04,354 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:04,354 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:10,342 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:10,344 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:10,371 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:10,372 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:10,374 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:14,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:14,371 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:14,371 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:20,384 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:20,384 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:20,410 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:20,411 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:20,411 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:24,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:24,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:24,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:30,421 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:30,421 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:30,438 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:30,441 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:30,441 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:34,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:34,415 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:34,415 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:40,451 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:40,453 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:40,476 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:40,477 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:40,479 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:44,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:44,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:44,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:50,489 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:50,489 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:34:50,513 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:34:50,516 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:34:50,516 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:34:54,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:34:54,456 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:34:54,456 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:00,526 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:00,526 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:00,551 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:00,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:00,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:04,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:04,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:04,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:10,566 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:10,566 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:10,586 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:10,589 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:10,589 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:14,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:14,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:14,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:20,604 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:20,605 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:20,622 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:20,625 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:20,625 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:24,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:24,502 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:24,502 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:30,637 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:30,638 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:30,658 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:30,661 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:30,663 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:34,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:34,517 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:34,517 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:40,674 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:40,674 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:40,700 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:40,703 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:40,703 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:44,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:44,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:44,539 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:50,716 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:50,716 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:35:50,730 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:35:50,733 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:35:50,733 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:35:54,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:35:54,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:35:54,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:00,744 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:00,746 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:36:00,763 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:00,766 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:00,766 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:04,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:04,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:04,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:10,778 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:10,778 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:36:10,796 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:10,798 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:10,798 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:14,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:14,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:14,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:20,809 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:20,809 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 56.65 2021-12-16 08:36:20,828 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:20,831 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:20,832 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:24,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.65 % 2021-12-16 08:36:24,608 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:24,608 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:30,842 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.34 % 2021-12-16 08:36:30,842 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.34 2021-12-16 08:36:30,867 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:30,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:30,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:34,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:36:34,631 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:34,631 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:40,881 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:36:40,881 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:36:40,905 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:40,907 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:40,907 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:44,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:36:44,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:44,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:50,917 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:36:50,917 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:36:50,936 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:36:50,937 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:36:50,937 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:36:54,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:36:54,664 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:36:54,664 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:00,948 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:00,948 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:00,967 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:00,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:00,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:04,676 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:04,686 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:04,686 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:10,986 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:10,986 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:11,003 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:11,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:11,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:14,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:14,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:14,717 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:21,017 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:21,017 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:21,042 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:21,045 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:21,045 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:24,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:24,734 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:24,734 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:31,055 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:31,055 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:31,073 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:31,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:31,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:34,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:34,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:34,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:41,085 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:41,085 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:41,101 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:41,102 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:41,102 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:44,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:44,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:44,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:51,114 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:51,114 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:37:51,134 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:37:51,137 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:37:51,137 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:37:54,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:37:54,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:37:54,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:01,151 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:01,151 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:01,176 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:01,178 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:01,178 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:04,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:04,806 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:04,806 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:11,190 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:11,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:11,219 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:11,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:11,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:14,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:14,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:14,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:21,230 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:21,230 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:21,247 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:21,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:21,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:24,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:24,838 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:24,839 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:31,263 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:31,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:31,280 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:31,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:31,282 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:34,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:34,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:34,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:41,295 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:41,295 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:41,319 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:41,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:41,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:44,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:44,877 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:44,878 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:51,332 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:51,332 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:38:51,348 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:38:51,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:38:51,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:38:54,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:38:54,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:38:54,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:01,362 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:01,364 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:39:01,388 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:01,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:01,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:04,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:04,913 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:04,913 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:11,404 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:11,404 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:39:11,423 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:11,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:11,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:14,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:14,931 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:14,931 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:21,437 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:21,437 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 57.57 2021-12-16 08:39:21,456 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:21,457 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:21,457 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:24,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.57 % 2021-12-16 08:39:24,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:24,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:31,467 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.03 % 2021-12-16 08:39:31,467 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.03 2021-12-16 08:39:31,484 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:31,486 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:31,486 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:34,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.27 % 2021-12-16 08:39:34,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:34,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:41,500 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:39:41,500 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:39:41,528 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:41,530 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:41,530 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:44,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:39:44,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:44,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:51,540 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:39:51,540 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:39:51,575 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:39:51,581 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:39:51,581 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:39:55,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:39:55,017 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:39:55,017 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:01,592 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:01,592 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:01,618 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:01,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:01,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:05,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:05,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:05,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:11,631 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:11,632 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:11,651 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:11,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:11,653 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:15,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:15,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:15,055 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:21,663 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:21,663 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:21,677 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:21,680 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:21,680 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:25,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:25,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:25,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:31,690 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:31,690 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:31,714 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:31,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:31,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:35,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:35,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:35,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:41,729 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:41,729 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:41,753 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:41,755 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:41,756 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:45,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:45,111 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:45,111 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:51,775 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:51,775 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:40:51,798 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:40:51,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:40:51,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:40:55,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:40:55,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:40:55,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:01,818 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:01,818 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:01,834 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:01,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:01,835 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:05,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:05,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:05,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:11,845 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:11,846 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:11,864 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:11,867 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:11,867 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:15,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:15,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:15,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:21,877 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:21,877 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:21,894 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:21,895 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:21,897 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:25,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:25,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:25,193 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:31,908 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:31,908 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:31,930 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:31,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:31,934 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:35,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:35,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:35,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:41,947 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:41,947 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:41,971 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:41,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:41,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:45,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:45,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:45,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:51,987 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:51,987 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:41:52,006 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:41:52,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:41:52,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:41:55,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:41:55,273 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:41:55,275 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:02,020 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:02,020 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:42:02,048 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:02,049 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:02,049 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:05,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:05,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:05,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:12,059 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:12,059 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:42:12,078 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:12,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:12,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:15,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:15,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:15,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:22,092 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:22,092 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.5 2021-12-16 08:42:22,118 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:22,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:22,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:25,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.50 % 2021-12-16 08:42:25,331 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:25,331 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:32,130 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.73 % 2021-12-16 08:42:32,130 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 58.73 2021-12-16 08:42:32,151 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:32,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:32,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:35,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.73 % 2021-12-16 08:42:35,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:35,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:42,167 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.19 % 2021-12-16 08:42:42,167 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.19 2021-12-16 08:42:42,181 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:42,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:42,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:45,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.19 % 2021-12-16 08:42:45,380 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:45,380 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:52,196 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:42:52,196 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:42:52,219 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:42:52,221 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:42:52,221 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:42:55,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:42:55,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:42:55,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:02,234 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:02,234 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:02,253 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:02,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:02,256 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:05,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:05,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:05,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:12,267 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:12,267 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:12,283 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:12,286 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:12,286 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:15,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:15,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:15,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:22,299 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:22,299 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:22,325 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:22,328 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:22,328 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:25,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:25,444 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:25,444 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:32,342 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:32,342 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:32,361 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:32,365 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:32,365 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:35,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:35,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:35,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:42,378 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:42,378 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:42,403 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:42,404 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:42,404 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:45,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:45,482 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:45,482 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:52,417 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:52,418 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:43:52,444 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:43:52,447 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:43:52,447 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:43:55,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:43:55,497 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:43:55,497 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:02,459 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:02,459 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:02,476 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:02,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:02,482 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:05,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:05,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:05,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:12,493 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:12,493 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:12,515 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:12,516 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:12,517 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:15,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:15,535 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:15,535 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:22,532 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:22,532 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:22,551 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:22,555 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:22,555 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:25,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:25,551 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:25,551 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:32,565 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:32,565 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:32,582 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:32,585 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:32,585 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:35,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:35,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:35,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:42,598 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:42,598 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:42,618 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:42,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:42,621 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:45,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:45,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:45,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:52,631 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:52,631 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:44:52,651 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:44:52,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:44:52,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:44:55,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:44:55,604 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:44:55,604 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:02,664 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:02,664 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:45:02,687 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:02,690 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:02,690 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:05,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:05,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:05,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:12,701 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:12,703 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:45:12,724 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:12,726 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:12,726 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:15,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:15,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:15,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:22,739 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:22,740 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:45:22,759 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:22,762 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:22,763 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:25,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:25,673 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:25,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:32,775 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:32,775 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.42 2021-12-16 08:45:32,799 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:32,801 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:32,801 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:35,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.42 % 2021-12-16 08:45:35,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:35,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:42,813 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.88 % 2021-12-16 08:45:42,815 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 59.88 2021-12-16 08:45:42,855 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:42,858 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:42,858 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:45,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.12 % 2021-12-16 08:45:45,709 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:45,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:52,874 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:45:52,874 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:45:52,894 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:45:52,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:45:52,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:45:55,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:45:55,730 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:45:55,730 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:02,913 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:02,913 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:02,930 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:02,931 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:02,931 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:05,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:05,759 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:05,759 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:12,946 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:12,946 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:12,967 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:12,969 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:12,969 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:15,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:15,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:15,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:22,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:22,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:23,006 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:23,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:23,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:25,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:25,802 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:25,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:33,020 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:33,022 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:33,042 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:33,043 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:33,045 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:35,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:35,826 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:35,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:43,059 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:43,059 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:43,089 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:43,092 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:43,092 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:45,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:45,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:45,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:53,105 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:53,105 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:46:53,121 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:46:53,122 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:46:53,122 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:46:55,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:46:55,881 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:46:55,881 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:03,135 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:03,137 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:03,161 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:03,163 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:03,163 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:05,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:05,901 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:05,901 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:13,176 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:13,177 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:13,194 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:13,198 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:13,198 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:15,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:15,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:15,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:23,211 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:23,211 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:23,237 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:23,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:23,240 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:25,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:25,947 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:25,947 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:33,255 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:33,255 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:33,273 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:33,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:33,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:35,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:35,964 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:35,964 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:43,290 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:43,290 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:43,309 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:43,312 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:43,312 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:45,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:45,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:45,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:53,328 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:53,328 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:47:53,355 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:47:53,357 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:47:53,357 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:47:55,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:47:56,006 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:47:56,006 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:03,369 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:03,369 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:48:03,390 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:03,392 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:03,392 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:06,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:06,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:06,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:13,403 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:13,404 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:48:13,418 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:13,421 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:13,421 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:16,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:16,046 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:16,046 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:23,433 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:23,433 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:48:23,460 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:23,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:23,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:26,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:26,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:26,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:33,473 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:33,474 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.35 2021-12-16 08:48:33,493 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:33,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:33,496 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:36,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.35 % 2021-12-16 08:48:36,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:36,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:43,506 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.58 % 2021-12-16 08:48:43,506 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 60.58 2021-12-16 08:48:43,525 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:43,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:43,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:46,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.58 % 2021-12-16 08:48:46,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:46,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:53,540 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.04 % 2021-12-16 08:48:53,540 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.04 2021-12-16 08:48:53,565 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:48:53,568 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:48:53,569 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:48:56,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.04 % 2021-12-16 08:48:56,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:48:56,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:03,581 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:03,581 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:03,598 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:03,601 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:03,601 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:06,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:06,141 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:06,141 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:13,612 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:13,614 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:13,637 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:13,638 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:13,638 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:16,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:16,173 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:16,173 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:23,651 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:23,653 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:23,674 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:23,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:23,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:26,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:26,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:26,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:33,688 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:33,688 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:33,704 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:33,707 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:33,707 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:36,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:36,206 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:36,206 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:43,720 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:43,720 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:43,746 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:43,749 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:43,749 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:46,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:46,229 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:46,229 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:53,759 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:53,759 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:49:53,778 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:49:53,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:49:53,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:49:56,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:49:56,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:49:56,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:03,792 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:03,792 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:03,815 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:03,816 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:03,816 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:06,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:06,266 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:06,266 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:13,828 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:13,828 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:13,848 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:13,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:13,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:16,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:16,292 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:23,861 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:23,861 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:23,881 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:23,884 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:23,884 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:26,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:26,309 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:26,311 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:33,898 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:33,898 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:33,914 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:33,915 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:33,915 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:36,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:36,326 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:36,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:43,928 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:43,928 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:43,953 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:43,954 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:43,956 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:46,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:46,346 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:46,348 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:53,967 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:53,969 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:50:53,986 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:50:53,989 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:50:53,989 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:50:56,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:50:56,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:50:56,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:04,000 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:04,000 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:51:04,022 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:04,026 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:04,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:06,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:06,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:06,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:14,039 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:14,040 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:51:14,058 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:14,059 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:14,059 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:16,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:16,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:16,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:24,072 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:24,072 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:51:24,099 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:24,104 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:24,104 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:26,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:26,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:26,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:34,115 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:34,115 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:51:34,154 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:34,158 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:34,158 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:36,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:36,436 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:36,437 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:44,171 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.27 % 2021-12-16 08:51:44,171 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.27 2021-12-16 08:51:44,194 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:44,197 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:44,197 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:46,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.50 % 2021-12-16 08:51:46,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:46,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:54,209 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.97 % 2021-12-16 08:51:54,210 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 61.97 2021-12-16 08:51:54,227 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:51:54,230 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:51:54,230 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:51:56,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.97 % 2021-12-16 08:51:56,469 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:51:56,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:04,243 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:04,243 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:04,266 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:04,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:04,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:06,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:06,489 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:06,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:14,282 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:14,282 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:14,301 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:14,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:14,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:16,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:16,507 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:16,507 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:24,315 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:24,316 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:24,335 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:24,338 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:24,339 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:26,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:26,529 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:26,530 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:34,351 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:34,351 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:34,368 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:34,371 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:34,371 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:36,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:36,545 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:36,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:44,381 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:44,382 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:44,408 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:44,411 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:44,411 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:46,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:46,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:46,563 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:54,421 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:54,423 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:52:54,440 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:52:54,444 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:52:54,444 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:52:56,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:52:56,584 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:52:56,584 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:04,456 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:04,456 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:04,473 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:04,474 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:04,474 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:06,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:06,599 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:06,599 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:14,486 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:14,486 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:14,505 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:14,506 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:14,506 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:16,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:16,615 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:16,615 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:24,519 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:24,519 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:24,536 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:24,539 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:24,539 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:26,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:26,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:26,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:34,552 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:34,552 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:34,569 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:34,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:34,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:36,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:36,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:36,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:44,585 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:44,585 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:44,608 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:44,608 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:44,609 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:46,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:46,667 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:46,667 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:54,621 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:54,621 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:53:54,638 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:53:54,641 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:53:54,641 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:53:56,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:53:56,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:53:56,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:04,653 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:04,653 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:04,677 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:04,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:04,678 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:06,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:06,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:06,709 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:14,690 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:14,690 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:14,710 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:14,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:14,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:16,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:16,727 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:16,727 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:24,724 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:24,724 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:24,740 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:24,743 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:24,743 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:26,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:26,750 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:26,750 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:34,755 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:34,755 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:34,779 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:34,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:34,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:36,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:36,767 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:36,767 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:44,801 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:44,801 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:44,826 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:44,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:44,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:46,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:46,782 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:46,782 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:54,841 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:54,842 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:54:54,865 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:54:54,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:54:54,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:54:56,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:54:56,799 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:54:56,799 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:04,880 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.20 % 2021-12-16 08:55:04,880 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.2 2021-12-16 08:55:04,895 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:04,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:04,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:06,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.43 % 2021-12-16 08:55:06,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:06,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:14,911 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.89 % 2021-12-16 08:55:14,913 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.89 2021-12-16 08:55:14,944 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:14,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:14,947 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:16,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.89 % 2021-12-16 08:55:16,841 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:16,841 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:24,960 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.89 % 2021-12-16 08:55:24,960 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 62.89 2021-12-16 08:55:24,994 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:24,999 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:24,999 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:26,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:26,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:26,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:35,009 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:35,009 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:55:35,026 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:35,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:35,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:36,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:36,884 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:36,884 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:45,040 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:45,040 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:55:45,066 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:45,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:45,069 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:46,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:46,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:46,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:55,079 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:55,079 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:55:55,098 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:55:55,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:55:55,101 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:55:56,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:55:56,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:55:56,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:05,112 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:05,114 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:05,128 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:05,131 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:05,131 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:06,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:06,943 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:06,943 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:15,141 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:15,141 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:15,160 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:15,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:15,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:16,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:16,959 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:16,959 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:25,184 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:25,184 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:25,207 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:25,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:25,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:26,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:26,980 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:26,980 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:35,221 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:35,221 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:35,240 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:35,242 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:35,242 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:36,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:37,002 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:37,002 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:45,253 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:45,255 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:45,273 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:45,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:45,276 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:47,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:47,016 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:47,016 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:55,288 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:55,288 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:56:55,305 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:56:55,309 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:56:55,309 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:56:57,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:56:57,036 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:56:57,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:05,321 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:05,321 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:05,345 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:05,348 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:05,348 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:07,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:07,059 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:07,059 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:15,368 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:15,368 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:15,387 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:15,390 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:15,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:17,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:17,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:17,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:25,401 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:25,401 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:25,427 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:25,428 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:25,428 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:27,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:27,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:27,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:35,441 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:35,441 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:35,457 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:35,460 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:35,460 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:37,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:37,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:37,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:45,490 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:45,490 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:45,509 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:45,513 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:45,513 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:47,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:47,134 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:47,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:55,523 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:55,523 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:57:55,542 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:57:55,545 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:57:55,545 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:57:57,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:57:57,153 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:57:57,153 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:05,555 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:58:05,555 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.12 2021-12-16 08:58:05,572 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:05,576 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:05,576 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:07,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.12 % 2021-12-16 08:58:07,174 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:07,174 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:15,588 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.35 % 2021-12-16 08:58:15,588 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.35 2021-12-16 08:58:15,611 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:15,614 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:15,614 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:17,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.35 % 2021-12-16 08:58:17,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:17,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:25,628 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.58 % 2021-12-16 08:58:25,628 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 63.58 2021-12-16 08:58:25,648 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:25,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:25,651 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:27,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.82 % 2021-12-16 08:58:27,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:27,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:35,664 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:35,665 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:58:35,683 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:35,684 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:35,686 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:37,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:37,246 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:37,246 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:45,696 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:45,697 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:58:45,711 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:45,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:45,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:47,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:47,283 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:47,285 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:55,724 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:55,724 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:58:55,755 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:58:55,759 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:58:55,759 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:58:57,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:58:57,299 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:58:57,299 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:05,769 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:05,769 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:05,786 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:05,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:05,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:07,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:07,315 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:07,315 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:15,802 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:15,803 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:15,826 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:15,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:15,829 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:17,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:17,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:17,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:25,841 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:25,841 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:25,859 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:25,862 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:25,862 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:27,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:27,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:27,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:35,877 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:35,877 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:35,898 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:35,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:35,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:37,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:37,384 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:37,384 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:45,914 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:45,914 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:45,937 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:45,941 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:45,941 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:47,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:47,401 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:47,401 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:55,954 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:55,954 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 08:59:55,980 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 08:59:55,983 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 08:59:55,983 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 08:59:57,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 08:59:57,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 08:59:57,418 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:05,993 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:05,993 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:06,010 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:06,013 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:06,013 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:07,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:07,440 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:07,441 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:16,023 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:16,023 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:16,043 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:16,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:16,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:17,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:17,460 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:17,460 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:26,058 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:26,059 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:26,073 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:26,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:26,076 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:27,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:27,477 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:27,477 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:36,088 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:36,088 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:36,105 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:36,108 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:36,108 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:37,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:37,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:37,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:46,118 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:46,118 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:46,137 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:46,140 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:46,141 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:47,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:47,519 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:47,519 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:56,151 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:56,153 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:00:56,177 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:00:56,180 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:00:56,180 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:00:57,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:00:57,539 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:00:57,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:06,190 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:01:06,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:01:06,206 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:06,207 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:06,209 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:07,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:01:07,555 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:07,555 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:16,220 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:01:16,220 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.05 2021-12-16 09:01:16,246 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:16,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:16,247 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:17,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 09:01:17,572 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:17,572 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:26,259 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.28 % 2021-12-16 09:01:26,259 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.28 2021-12-16 09:01:26,282 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:26,285 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:26,285 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:27,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.28 % 2021-12-16 09:01:27,589 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:27,591 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:36,299 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.51 % 2021-12-16 09:01:36,299 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.51 2021-12-16 09:01:36,319 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:36,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:36,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:37,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.51 % 2021-12-16 09:01:37,607 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:37,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:46,335 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.74 % 2021-12-16 09:01:46,335 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.74 2021-12-16 09:01:46,355 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:46,358 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:46,358 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:47,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:01:47,622 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:47,622 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:56,368 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:01:56,369 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:01:56,397 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:01:56,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:01:56,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:01:57,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:01:57,647 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:01:57,647 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:06,410 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:06,410 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:06,430 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:06,436 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:06,436 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:07,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:07,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:07,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:16,447 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:16,448 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:16,473 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:16,474 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:16,474 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:17,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:17,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:17,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:26,487 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:26,489 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:26,516 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:26,519 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:26,519 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:27,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:27,700 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:27,700 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:36,529 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:36,530 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:36,549 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:36,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:36,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:37,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:37,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:37,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:46,562 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:46,563 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:46,588 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:46,589 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:46,589 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:47,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:47,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:47,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:56,601 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:56,601 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:02:56,628 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:02:56,632 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:02:56,632 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:02:57,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:02:57,756 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:02:57,757 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:06,644 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:06,644 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:06,668 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:06,671 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:06,671 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:07,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:07,772 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:07,772 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:16,684 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:16,684 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:16,707 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:16,710 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:16,710 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:17,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:17,788 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:17,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:26,723 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:26,724 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:26,746 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:26,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:26,750 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:27,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:27,811 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:27,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:36,766 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:36,766 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:36,786 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:36,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:36,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:37,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:37,834 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:37,834 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:46,799 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:46,799 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:46,821 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:46,825 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:46,825 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:47,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:47,857 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:47,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:56,836 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:56,836 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:03:56,855 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:03:56,858 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:03:56,858 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:03:57,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:03:57,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:03:57,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:06,868 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:06,868 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:04:06,884 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:06,888 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:06,888 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:07,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:07,892 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:07,892 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:16,898 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:16,898 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:04:16,918 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:16,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:16,923 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:17,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:17,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:17,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:26,950 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:26,950 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 64.97 2021-12-16 09:04:26,970 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:26,971 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:26,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:27,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.97 % 2021-12-16 09:04:27,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:27,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:36,982 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.20 % 2021-12-16 09:04:36,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.2 2021-12-16 09:04:36,997 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:37,000 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:37,000 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:37,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.20 % 2021-12-16 09:04:37,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:37,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:47,010 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.43 % 2021-12-16 09:04:47,010 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.43 2021-12-16 09:04:47,028 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:47,030 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:47,030 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:47,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.43 % 2021-12-16 09:04:47,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:47,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:57,042 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.66 % 2021-12-16 09:04:57,042 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.66 2021-12-16 09:04:57,059 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:04:57,062 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:04:57,062 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:04:57,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:04:57,990 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:04:57,990 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:07,073 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:07,073 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:07,092 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:07,095 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:07,095 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:08,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:08,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:08,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:17,117 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:17,118 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:17,134 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:17,138 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:17,138 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:18,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:18,029 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:18,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:27,148 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:27,148 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:27,168 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:27,171 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:27,171 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:28,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:28,049 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:28,049 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:37,184 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:37,184 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:37,203 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:37,204 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:37,204 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:38,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:38,065 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:38,065 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:47,214 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:47,214 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:47,232 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:47,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:47,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:48,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:48,088 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:48,088 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:57,247 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:57,247 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:05:57,266 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:05:57,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:05:57,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:05:58,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:05:58,104 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:05:58,104 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:07,282 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:07,282 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:07,299 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:07,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:07,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:08,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:08,118 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:08,118 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:17,315 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:17,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:17,332 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:17,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:17,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:18,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:18,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:18,132 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:27,352 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:27,352 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:27,375 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:27,380 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:27,380 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:28,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:28,150 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:28,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:37,391 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:37,391 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:37,411 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:37,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:37,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:38,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:38,170 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:38,170 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:47,426 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:47,426 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:47,451 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:47,453 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:47,453 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:48,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:48,193 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:48,194 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:57,469 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:57,469 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:06:57,490 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:06:57,493 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:06:57,493 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:06:58,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:06:58,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:06:58,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:07,505 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:07,505 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:07:07,520 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:07,523 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:07,523 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:08,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:08,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:08,239 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:17,535 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:17,535 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:07:17,558 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:17,559 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:17,559 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:18,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:18,253 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:18,253 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:27,571 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:27,571 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:07:27,592 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:27,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:27,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:28,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:28,269 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:28,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:37,607 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:37,607 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 65.9 2021-12-16 09:07:37,625 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:37,627 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:37,627 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:38,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.90 % 2021-12-16 09:07:38,289 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:38,289 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:47,638 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.13 % 2021-12-16 09:07:47,638 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.13 2021-12-16 09:07:47,663 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:47,665 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:47,665 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:48,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.13 % 2021-12-16 09:07:48,305 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:48,305 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:57,677 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.36 % 2021-12-16 09:07:57,677 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.36 2021-12-16 09:07:57,696 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:07:57,698 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:07:57,698 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:07:58,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.59 % 2021-12-16 09:07:58,322 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:07:58,322 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:07,710 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:07,710 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:07,733 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:07,736 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:07,736 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:08,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:08,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:08,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:17,747 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:17,747 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:17,769 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:17,773 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:17,773 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:18,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:18,361 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:18,361 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:27,782 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:27,783 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:27,799 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:27,802 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:27,803 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:28,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:28,385 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:28,385 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:37,825 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:37,825 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:37,848 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:37,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:37,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:38,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:38,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:38,407 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:47,867 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:47,868 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:47,888 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:47,891 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:47,891 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:48,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:48,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:48,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:57,904 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:57,904 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:08:57,918 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:08:57,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:08:57,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:08:58,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:08:58,450 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:08:58,450 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:07,931 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:07,931 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:07,957 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:07,960 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:07,960 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:08,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:08,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:08,467 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:17,970 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:17,970 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:17,989 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:17,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:17,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:18,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:18,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:18,486 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:28,003 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:28,005 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:28,026 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:28,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:28,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:28,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:28,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:28,516 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:38,042 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:38,042 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:38,069 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:38,072 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:38,072 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:38,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:38,532 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:38,532 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:48,082 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:48,082 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:48,107 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:48,109 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:48,111 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:48,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:48,548 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:48,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:58,121 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:58,121 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:09:58,147 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:09:58,148 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:09:58,148 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:09:58,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:09:58,571 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:09:58,571 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:08,160 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:08,161 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:08,188 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:08,194 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:08,194 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:08,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:08,588 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:08,588 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:18,204 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:18,204 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:18,223 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:18,224 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:18,224 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:18,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:18,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:18,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:28,236 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:28,237 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:28,260 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:28,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:28,263 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:28,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:28,628 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:28,628 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:38,275 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:38,276 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:38,299 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:38,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:38,302 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:38,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:38,650 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:38,650 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:48,312 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:48,312 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:48,342 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:48,345 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:48,346 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:48,658 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:48,665 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:48,665 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:58,357 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:58,357 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 66.82 2021-12-16 09:10:58,375 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:10:58,377 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:10:58,377 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:10:58,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.82 % 2021-12-16 09:10:58,684 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:10:58,684 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:08,388 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.05 % 2021-12-16 09:11:08,390 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.05 2021-12-16 09:11:08,414 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:08,417 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:08,417 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:08,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.05 % 2021-12-16 09:11:08,707 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:08,707 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:18,434 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.28 % 2021-12-16 09:11:18,434 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.28 2021-12-16 09:11:18,460 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:18,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:18,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:18,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.28 % 2021-12-16 09:11:18,729 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:18,729 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:28,476 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:28,477 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:11:28,496 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:28,499 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:28,499 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:28,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:28,746 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:28,746 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:38,510 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:38,510 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:11:38,528 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:38,529 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:38,529 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:38,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:38,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:38,763 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:48,543 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:48,543 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:11:48,562 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:48,565 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:48,565 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:48,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:48,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:48,788 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:58,576 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:58,576 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:11:58,594 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:11:58,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:11:58,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:11:58,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:11:58,803 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:11:58,803 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:08,608 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:08,608 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:08,631 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:08,634 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:08,634 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:08,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:08,823 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:08,823 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:18,645 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:18,647 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:18,664 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:18,667 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:18,667 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:18,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:18,842 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:18,842 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:28,677 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:28,677 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:28,698 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:28,700 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:28,700 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:28,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:28,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:28,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:38,711 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:38,711 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:38,736 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:38,739 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:38,739 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:38,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:38,878 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:38,880 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:48,747 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:48,749 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:48,773 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:48,776 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:48,776 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:48,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:48,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:48,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:58,788 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:58,788 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:12:58,811 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:12:58,812 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:12:58,812 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:12:58,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:12:58,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:12:58,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:08,822 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:08,823 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:08,848 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:08,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:08,851 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:08,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:08,937 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:08,937 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:18,861 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:18,861 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:18,897 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:18,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:18,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:18,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:18,953 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:18,953 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:28,913 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:28,913 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:28,937 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:28,938 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:28,938 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:28,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:28,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:28,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:38,950 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:38,950 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:38,979 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:38,982 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:38,983 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:38,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:38,997 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:38,997 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:48,993 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:48,993 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:49,010 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:49,012 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:49,016 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:49,016 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:49,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:49,022 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:59,028 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:59,028 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:13:59,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:13:59,042 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:13:59,042 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:13:59,046 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:13:59,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:13:59,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:09,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:14:09,065 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.75 % 2021-12-16 09:14:09,065 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.75 2021-12-16 09:14:09,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:09,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:09,095 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:09,099 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:09,099 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:19,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.98 % 2021-12-16 09:14:19,084 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:19,084 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:19,111 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.98 % 2021-12-16 09:14:19,111 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 67.98 2021-12-16 09:14:19,125 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:19,128 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:19,128 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:29,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.21 % 2021-12-16 09:14:29,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:29,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:29,140 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.21 % 2021-12-16 09:14:29,140 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.21 2021-12-16 09:14:29,165 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:29,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:29,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:39,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:39,132 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:39,134 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:39,178 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:39,178 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:14:39,197 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:39,200 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:39,200 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:49,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:49,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:49,148 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:49,211 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:49,211 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:14:49,233 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:49,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:49,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:59,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:59,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:14:59,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:14:59,247 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:14:59,247 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:14:59,269 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:14:59,272 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:14:59,272 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:09,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:09,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:09,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:09,283 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:09,283 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:09,312 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:09,315 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:09,315 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:19,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:19,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:19,214 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:19,326 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:19,326 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:19,342 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:19,344 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:19,344 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:29,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:29,236 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:29,236 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:29,357 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:29,357 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:29,380 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:29,382 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:29,382 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:39,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:39,262 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:39,263 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:39,394 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:39,394 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:39,411 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:39,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:39,414 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:49,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:49,280 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:49,280 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:49,424 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:49,424 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:49,450 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:49,451 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:49,451 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:59,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:59,303 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:15:59,303 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:15:59,460 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:15:59,461 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:15:59,477 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:15:59,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:15:59,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:09,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:09,325 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:09,325 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:09,490 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:09,490 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:09,506 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:09,507 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:09,509 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:19,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:19,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:19,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:19,520 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:19,522 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:19,549 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:19,551 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:19,551 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:29,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:29,368 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:29,368 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:29,562 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:29,562 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:29,582 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:29,585 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:29,585 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:39,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:39,391 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:39,391 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:39,596 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:39,596 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:39,621 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:39,625 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:39,625 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:49,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:49,408 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:49,410 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:49,638 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:49,638 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:49,663 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:49,665 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:49,665 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:59,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:59,430 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:16:59,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:16:59,683 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:16:59,683 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:16:59,700 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:16:59,703 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:16:59,703 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:09,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:17:09,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:09,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:09,713 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:17:09,713 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:17:09,733 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:09,734 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:09,734 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:19,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:17:19,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:19,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:19,746 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.67 % 2021-12-16 09:17:19,746 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.67 2021-12-16 09:17:19,766 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:19,770 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:19,770 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:29,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.90 % 2021-12-16 09:17:29,486 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:29,487 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:29,782 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.90 % 2021-12-16 09:17:29,782 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 68.9 2021-12-16 09:17:29,798 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:29,799 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:29,799 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:39,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.13 % 2021-12-16 09:17:39,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:39,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:39,811 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.36 % 2021-12-16 09:17:39,811 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.36 2021-12-16 09:17:39,835 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:39,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:39,838 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:49,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:17:49,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:49,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:49,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:17:49,849 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:17:49,869 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:49,875 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:49,875 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:59,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:17:59,540 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:17:59,540 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:17:59,890 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:17:59,890 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:17:59,914 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:17:59,915 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:17:59,915 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:09,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:09,558 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:09,558 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:09,927 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:09,928 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:18:09,953 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:18:09,956 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:18:09,956 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:19,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:19,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:19,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:19,967 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:19,967 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:18:19,989 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:18:19,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:18:19,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:29,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:29,602 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:29,602 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:30,003 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:30,003 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:18:30,028 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:18:30,029 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:18:30,029 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:39,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:39,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:39,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:40,056 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:40,058 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:18:40,078 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:18:40,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:18:40,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:49,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:49,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:49,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:50,092 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:50,092 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:18:50,109 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:18:50,111 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:18:50,111 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:18:59,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:18:59,655 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:18:59,655 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:00,122 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:00,122 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:00,150 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:00,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:00,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:09,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:09,687 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:09,687 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:10,167 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:10,167 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:10,187 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:10,188 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:10,190 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:19,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:19,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:19,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:20,201 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:20,203 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:20,229 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:20,230 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:20,230 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:29,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:29,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:29,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:30,242 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:30,242 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:30,263 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:30,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:30,266 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:39,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:39,742 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:39,742 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:40,276 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:40,278 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:40,293 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:40,296 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:40,296 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:49,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:49,766 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:49,766 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:50,308 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:50,308 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:19:50,332 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:19:50,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:19:50,335 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:19:59,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:19:59,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:19:59,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:00,345 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:00,346 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:00,365 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:00,368 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:00,368 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:09,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:09,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:09,811 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:10,378 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:10,378 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:10,394 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:10,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:10,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:19,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:19,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:19,828 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:20,414 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:20,414 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:20,440 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:20,443 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:20,443 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:29,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:29,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:29,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:30,456 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:30,456 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:30,480 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:30,483 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:30,483 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:39,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:39,877 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:39,877 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:40,496 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:40,496 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:40,523 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:40,525 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:40,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:49,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:49,898 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:49,898 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:50,536 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:50,536 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:20:50,559 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:20:50,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:20:50,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:20:59,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:20:59,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:20:59,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:00,575 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.60 % 2021-12-16 09:21:00,575 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 69.6 2021-12-16 09:21:00,598 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:00,602 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:00,602 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:09,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:09,930 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:21:09,930 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:10,615 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:10,615 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:21:10,644 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:10,647 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:10,648 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:19,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:19,948 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:21:19,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:20,657 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:20,658 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:21:20,674 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:20,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:20,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:29,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:29,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:21:29,971 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:30,690 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:30,690 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:21:30,714 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:30,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:30,717 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:39,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:39,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:21:39,986 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:40,730 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:40,730 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:21:40,753 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:40,756 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:40,756 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:49,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:50,000 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:21:50,000 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:21:50,769 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:21:50,769 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:21:50,793 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:21:50,796 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:21:50,796 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:00,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:00,019 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:00,019 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:00,806 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:00,808 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:00,825 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:00,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:00,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:10,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:10,032 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:10,033 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:10,838 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:10,839 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:10,868 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:10,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:10,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:20,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:20,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:20,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:20,882 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:20,882 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:20,900 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:20,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:20,901 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:30,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:30,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:30,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:30,924 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:30,924 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:30,941 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:30,946 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:30,946 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:40,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:40,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:40,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:40,957 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:40,957 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:40,973 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:40,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:40,976 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:50,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:50,107 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:22:50,107 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:22:50,989 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:22:50,990 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:22:51,007 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:22:51,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:22:51,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:00,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:00,125 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:00,125 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:01,019 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:01,020 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:01,038 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:01,040 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:01,040 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:10,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:10,144 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:10,144 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:11,052 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:11,052 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:11,079 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:11,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:11,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:20,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:20,167 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:20,167 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:21,094 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:21,095 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:21,115 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:21,118 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:21,118 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:30,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:30,183 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:30,183 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:31,128 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:31,128 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:31,144 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:31,148 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:31,148 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:40,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:40,200 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:40,200 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:41,158 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:41,158 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:41,174 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:41,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:41,177 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:50,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:50,223 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:23:50,224 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:23:51,188 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:23:51,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:23:51,213 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:23:51,217 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:23:51,217 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:00,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:24:00,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:00,242 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:01,229 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:24:01,230 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:24:01,247 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:01,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:01,252 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:10,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:24:10,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:10,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:11,263 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.52 % 2021-12-16 09:24:11,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 70.52 2021-12-16 09:24:11,288 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:11,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:11,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:20,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:20,282 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:20,282 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:21,301 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:21,302 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:24:21,319 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:21,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:21,322 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:30,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:30,296 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:30,296 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:31,332 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:31,332 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:24:31,349 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:31,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:31,351 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:40,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:40,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:40,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:41,365 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:41,365 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:24:41,388 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:41,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:41,391 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:50,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:50,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:24:50,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:24:51,403 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:24:51,403 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:24:51,437 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:24:51,440 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:24:51,440 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:00,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:00,345 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:00,345 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:01,451 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:01,451 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:01,473 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:01,476 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:01,476 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:10,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:10,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:10,362 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:11,489 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:11,489 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:11,509 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:11,512 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:11,512 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:20,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:20,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:20,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:21,523 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:21,523 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:21,543 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:21,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:21,546 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:30,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:30,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:30,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:31,556 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:31,558 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:31,579 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:31,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:31,584 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:40,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:40,410 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:40,411 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:41,595 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:41,595 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:41,612 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:41,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:41,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:50,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:50,428 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:25:50,428 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:25:51,627 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:25:51,628 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:25:51,644 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:25:51,644 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:25:51,645 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:00,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:00,453 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:00,453 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:01,655 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:01,655 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:01,674 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:01,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:01,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:10,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:10,470 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:10,470 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:11,690 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:11,690 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:11,713 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:11,716 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:11,716 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:20,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:20,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:20,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:21,726 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:21,727 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:21,750 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:21,753 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:21,753 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:30,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:30,509 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:30,509 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:31,763 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:31,763 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:31,782 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:31,785 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:31,785 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:40,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:40,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:40,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:41,801 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:41,801 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:41,818 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:41,821 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:41,821 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:50,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:50,549 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:26:50,549 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:26:51,832 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:26:51,832 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:26:51,851 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:26:51,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:26:51,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:00,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:00,565 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:00,565 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:01,867 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:01,867 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:27:01,891 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:01,892 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:01,892 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:10,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:10,582 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:10,582 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:11,911 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:11,911 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:27:11,928 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:11,933 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:11,933 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:20,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:20,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:20,607 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:21,944 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:21,944 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.45 2021-12-16 09:27:21,969 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:21,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:21,970 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:30,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.45 % 2021-12-16 09:27:30,621 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:30,621 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:31,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.68 % 2021-12-16 09:27:31,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 71.68 2021-12-16 09:27:32,003 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:32,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:32,009 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:40,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:27:40,644 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:40,644 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:42,017 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:27:42,019 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:27:42,033 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:42,036 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:42,036 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:50,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:27:50,658 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:27:50,660 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:27:52,048 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:27:52,049 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:27:52,072 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:27:52,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:27:52,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:00,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:00,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:00,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:02,085 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:02,085 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:02,102 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:02,107 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:02,107 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:10,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:10,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:10,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:12,118 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:12,118 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:12,141 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:12,142 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:12,142 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:20,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:20,713 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:20,713 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:22,154 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:22,154 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:22,181 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:22,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:22,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:30,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:30,753 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:30,753 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:32,196 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:32,196 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:32,216 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:32,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:32,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:40,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:40,790 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:40,790 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:42,230 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:42,230 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:42,252 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:42,255 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:42,255 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:50,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:50,808 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:28:50,808 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:28:52,265 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:28:52,265 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:28:52,282 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:28:52,285 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:28:52,285 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:00,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:00,828 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:00,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:02,295 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:02,295 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:02,311 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:02,312 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:02,312 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:10,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:10,845 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:10,845 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:12,323 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:12,323 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:12,348 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:12,349 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:12,349 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:20,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:20,861 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:20,861 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:22,365 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:22,365 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:22,385 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:22,388 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:22,390 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:30,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:30,888 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:30,888 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:32,400 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:32,400 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:32,417 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:32,420 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:32,420 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:40,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:40,904 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:40,904 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:42,431 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:42,431 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:42,457 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:42,460 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:42,460 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:50,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:50,920 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:29:50,920 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:29:52,469 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:29:52,470 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:29:52,486 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:29:52,490 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:29:52,490 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:00,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:00,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:00,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:02,500 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:02,502 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:30:02,525 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:02,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:02,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:10,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:10,956 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:10,957 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:12,539 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:12,539 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:30:12,559 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:12,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:12,562 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:20,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:20,973 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:20,973 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:22,573 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:22,573 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:30:22,592 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:22,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:22,595 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:30,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:30,992 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:30,992 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:32,608 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:32,608 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:30:32,632 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:32,634 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:32,634 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:41,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:41,007 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:41,007 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:42,664 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.37 % 2021-12-16 09:30:42,664 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 72.37 2021-12-16 09:30:42,683 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:42,686 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:42,686 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:51,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.60 % 2021-12-16 09:30:51,035 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:30:51,036 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:30:52,697 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:30:52,697 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:30:52,720 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:30:52,721 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:30:52,721 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:01,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:01,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:01,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:02,733 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:02,733 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:02,759 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:02,762 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:02,762 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:11,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:11,066 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:11,066 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:12,773 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:12,773 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:12,803 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:12,805 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:12,806 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:21,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:21,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:21,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:22,816 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:22,816 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:22,841 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:22,842 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:22,842 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:31,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:31,099 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:31,099 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:32,852 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:32,854 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:32,877 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:32,880 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:32,880 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:41,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:41,115 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:41,115 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:42,897 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:42,897 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:42,926 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:42,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:42,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:51,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:51,130 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:31:51,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:31:52,943 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:31:52,943 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:31:52,960 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:31:52,963 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:31:52,963 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:01,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:01,154 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:01,154 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:02,974 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:02,974 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:03,000 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:03,002 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:03,002 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:11,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:11,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:11,171 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:13,015 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:13,015 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:13,033 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:13,036 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:13,038 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:21,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:21,188 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:21,188 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:23,049 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:23,051 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:23,073 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:23,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:23,075 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:31,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:31,210 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:31,210 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:33,088 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:33,088 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:33,112 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:33,115 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:33,115 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:41,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:41,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:41,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:43,130 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:43,130 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:43,150 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:43,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:43,154 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:51,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:51,244 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:32:51,244 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:32:53,165 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:32:53,165 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:32:53,181 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:32:53,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:32:53,184 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:01,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:01,263 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:01,265 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:03,194 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:03,196 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:03,217 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:03,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:03,236 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:11,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:11,292 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:11,293 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:13,247 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:13,249 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:13,267 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:13,270 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:13,272 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:21,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:21,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:21,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:23,280 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:23,280 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:23,296 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:23,299 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:23,299 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:31,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:31,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:31,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:33,313 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:33,315 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:33,336 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:33,342 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:33,342 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:41,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:41,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:41,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:43,351 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:43,351 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:43,368 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:43,369 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:43,371 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:51,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:51,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:33:51,359 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:33:53,381 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:33:53,381 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:33:53,395 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:33:53,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:33:53,398 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:01,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:34:01,377 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:01,377 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:03,407 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:34:03,407 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.29 2021-12-16 09:34:03,424 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:03,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:03,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:11,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.29 % 2021-12-16 09:34:11,395 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:11,395 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:13,437 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.53 % 2021-12-16 09:34:13,437 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 73.53 2021-12-16 09:34:13,466 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:13,469 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:13,469 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:21,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.99 % 2021-12-16 09:34:21,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:21,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:23,480 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:23,480 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:34:23,496 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:23,497 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:23,497 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:31,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:31,431 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:31,431 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:33,510 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:33,510 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:34:33,529 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:33,532 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:33,533 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:41,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:41,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:41,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:43,545 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:43,546 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:34:43,571 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:43,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:43,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:51,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:51,466 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:34:51,466 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:34:53,582 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:34:53,584 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:34:53,608 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:34:53,612 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:34:53,615 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:01,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:01,490 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:01,490 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:03,631 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:03,631 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:03,661 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:03,667 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:03,668 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:11,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:11,506 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:11,506 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:13,680 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:13,680 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:13,704 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:13,707 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:13,707 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:21,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:21,522 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:21,522 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:23,720 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:23,720 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:23,747 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:23,752 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:23,752 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:31,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:31,542 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:31,543 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:33,767 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:33,767 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:33,788 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:33,790 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:33,790 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:41,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:41,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:41,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:43,802 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:43,802 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:43,826 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:43,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:43,828 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:51,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:51,581 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:35:51,581 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:35:53,842 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:35:53,842 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:35:53,861 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:35:53,865 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:35:53,865 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:01,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:01,598 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:01,598 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:03,877 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:03,878 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:03,894 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:03,897 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:03,897 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:11,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:11,614 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:11,614 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:13,910 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:13,910 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:13,927 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:13,928 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:13,930 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:21,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:21,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:21,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:23,964 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:23,964 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:23,986 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:23,989 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:23,989 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:31,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:31,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:31,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:34,000 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:34,000 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:34,026 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:34,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:34,028 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:41,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:41,680 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:41,680 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:44,039 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:44,039 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:44,062 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:44,063 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:44,065 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:51,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:51,726 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:36:51,726 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:36:54,075 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:36:54,075 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:36:54,095 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:36:54,099 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:36:54,099 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:01,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:01,743 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:01,743 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:04,111 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:04,111 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:37:04,137 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:04,140 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:04,140 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:11,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:11,760 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:11,760 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:14,160 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:14,160 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:37:14,184 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:14,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:14,187 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:21,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:21,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:21,786 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:24,197 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.22 % 2021-12-16 09:37:24,197 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.22 2021-12-16 09:37:24,213 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:24,213 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:24,213 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:31,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.45 % 2021-12-16 09:37:31,801 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:31,801 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:34,224 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.45 % 2021-12-16 09:37:34,224 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 74.45 2021-12-16 09:37:34,247 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:34,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:34,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:41,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:37:41,815 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:41,816 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:44,263 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:37:44,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:37:44,279 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:44,283 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:44,283 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:51,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:37:51,836 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:37:51,836 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:37:54,296 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:37:54,296 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:37:54,319 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:37:54,321 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:37:54,321 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:01,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:01,852 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:01,854 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:04,332 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:04,332 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:04,352 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:04,355 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:04,355 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:11,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:11,887 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:11,887 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:14,368 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:14,368 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:14,387 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:14,390 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:14,390 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:21,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:21,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:21,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:24,401 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:24,401 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:24,426 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:24,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:24,427 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:31,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:31,933 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:31,933 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:34,456 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:34,457 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:34,476 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:34,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:34,480 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:41,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:41,950 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:41,950 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:44,493 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:44,493 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:44,510 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:44,512 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:44,513 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:51,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:51,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:38:51,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:38:54,523 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:38:54,523 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:38:54,549 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:38:54,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:38:54,552 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:01,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:01,987 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:01,987 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:04,562 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:04,562 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:04,585 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:04,588 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:04,589 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:12,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:12,010 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:12,012 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:14,599 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:14,599 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:14,615 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:14,618 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:14,618 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:22,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:22,028 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:22,029 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:24,630 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:24,631 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:24,651 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:24,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:24,654 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:32,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:32,048 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:32,048 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:34,667 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:34,667 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:34,684 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:34,687 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:34,687 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:42,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:42,069 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:42,069 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:44,698 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:44,698 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:44,723 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:44,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:44,724 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:52,081 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:52,086 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:39:52,086 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:39:54,736 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:39:54,736 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:39:54,755 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:39:54,757 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:39:54,757 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:02,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:02,108 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:02,108 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:04,769 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:04,769 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:04,786 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:04,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:04,789 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:12,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:12,128 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:12,130 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:14,803 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:14,803 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:14,835 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:14,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:14,836 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:22,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:22,145 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:22,147 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:24,848 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:24,848 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:24,865 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:24,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:24,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:32,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:32,164 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:32,164 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:34,878 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:34,880 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:34,895 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:34,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:34,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:42,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:42,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:42,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:44,915 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:44,915 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:44,936 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:44,938 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:44,938 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:52,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:52,203 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:40:52,203 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:40:54,950 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:40:54,950 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:40:54,970 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:40:54,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:40:54,973 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:02,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:41:02,227 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:02,227 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:04,986 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.14 % 2021-12-16 09:41:04,986 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.14 2021-12-16 09:41:05,005 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:05,006 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:05,007 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:12,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.38 % 2021-12-16 09:41:12,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:12,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:15,022 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.38 % 2021-12-16 09:41:15,023 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 75.38 2021-12-16 09:41:15,049 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:15,052 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:15,052 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:22,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.84 % 2021-12-16 09:41:22,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:22,269 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:25,062 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:25,062 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:41:25,081 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:25,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:25,082 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:32,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:32,286 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:32,288 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:35,095 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:35,095 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:41:35,125 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:35,128 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:35,128 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:42,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:42,302 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:42,302 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:45,144 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:45,144 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:41:45,164 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:45,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:45,167 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:52,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:52,321 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:41:52,321 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:41:55,177 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:41:55,177 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:41:55,197 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:41:55,197 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:41:55,198 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:02,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:02,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:02,342 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:05,210 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:05,210 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:05,230 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:05,233 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:05,233 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:12,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:12,358 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:12,358 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:15,246 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:15,246 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:15,266 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:15,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:15,269 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:22,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:22,375 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:22,375 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:25,282 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:25,282 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:25,302 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:25,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:25,303 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:32,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:32,407 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:32,408 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:35,322 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:35,322 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:35,358 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:35,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:35,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:42,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:42,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:42,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:45,371 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:45,372 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:45,391 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:45,394 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:45,395 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:52,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:52,447 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:42:52,447 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:42:55,407 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:42:55,407 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:42:55,431 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:42:55,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:42:55,434 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:02,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:02,464 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:02,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:05,444 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:05,444 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:05,463 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:05,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:05,467 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:12,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:12,483 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:12,483 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:15,483 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:15,483 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:15,509 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:15,510 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:15,510 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:22,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:22,503 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:22,503 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:25,523 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:25,523 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:25,552 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:25,555 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:25,555 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:32,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:32,520 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:32,520 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:35,569 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:35,569 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:35,585 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:35,586 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:35,586 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:42,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:42,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:42,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:45,598 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:45,598 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:45,625 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:45,628 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:45,630 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:52,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:52,569 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:43:52,569 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:43:55,641 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:43:55,641 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:43:55,658 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:43:55,661 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:43:55,661 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:02,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:02,585 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:02,585 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:05,674 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:05,674 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:44:05,698 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:05,700 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:05,700 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:12,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:12,605 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:12,605 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:15,713 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:15,713 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:44:15,733 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:15,736 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:15,737 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:22,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:22,619 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:22,619 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:25,752 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.07 % 2021-12-16 09:44:25,752 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.07 2021-12-16 09:44:25,778 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:25,780 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:25,782 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:32,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.30 % 2021-12-16 09:44:32,635 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:32,635 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:35,796 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.30 % 2021-12-16 09:44:35,796 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.3 2021-12-16 09:44:35,818 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:35,819 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:35,819 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:42,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.53 % 2021-12-16 09:44:42,657 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:42,658 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:45,838 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:44:45,838 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:44:45,864 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:45,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:45,868 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:52,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:44:52,674 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:44:52,674 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:44:55,878 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:44:55,880 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:44:55,897 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:44:55,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:44:55,898 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:02,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:02,690 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:02,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:05,910 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:05,910 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:05,934 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:05,937 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:05,937 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:12,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:12,716 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:12,716 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:15,947 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:15,948 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:15,967 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:15,971 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:15,971 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:22,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:22,733 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:22,733 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:25,983 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:25,983 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:26,002 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:26,003 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:26,003 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:32,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:32,752 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:32,752 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:36,019 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:36,019 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:36,046 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:36,049 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:36,049 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:42,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:42,792 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:42,792 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:46,059 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:46,059 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:46,075 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:46,078 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:46,078 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:52,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:52,809 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:45:52,809 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:45:56,088 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:45:56,088 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:45:56,112 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:45:56,115 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:45:56,115 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:02,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:02,829 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:02,829 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:06,127 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:06,128 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:06,148 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:06,151 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:06,151 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:12,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:12,851 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:12,851 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:16,164 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:16,164 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:16,180 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:16,181 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:16,181 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:22,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:22,865 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:22,865 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:26,193 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:26,194 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:26,217 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:26,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:26,220 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:32,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:32,900 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:32,900 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:36,233 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:36,233 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:36,256 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:36,260 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:36,260 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:42,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:42,914 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:42,914 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:46,273 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:46,273 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:46,295 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:46,298 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:46,298 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:52,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:52,934 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:46:52,934 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:46:56,318 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:46:56,319 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:46:56,342 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:46:56,346 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:46:56,346 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:02,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:02,970 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:02,970 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:06,358 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:06,359 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:47:06,382 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:06,387 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:06,388 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:12,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:12,983 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:12,983 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:16,400 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:16,400 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:47:16,420 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:16,423 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:16,423 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:23,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:23,023 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:23,023 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:26,434 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:26,434 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:47:26,451 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:26,453 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:26,454 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:33,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:33,039 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:33,039 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:36,464 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:36,466 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 76.99 2021-12-16 09:47:36,486 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:36,489 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:36,490 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:43,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.99 % 2021-12-16 09:47:43,053 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:43,053 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:46,502 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.23 % 2021-12-16 09:47:46,502 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.23 2021-12-16 09:47:46,523 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:46,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:46,526 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:53,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.23 % 2021-12-16 09:47:53,075 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:47:53,075 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:47:56,546 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.23 % 2021-12-16 09:47:56,546 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.23 2021-12-16 09:47:56,563 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:47:56,565 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:47:56,565 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:03,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:03,091 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:03,091 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:06,576 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:06,576 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:06,595 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:06,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:06,596 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:13,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:13,109 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:13,109 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:16,612 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:16,614 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:16,632 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:16,637 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:16,637 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:23,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:23,127 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:23,127 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:26,648 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:26,648 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:26,674 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:26,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:26,677 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:33,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:33,142 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:33,142 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:36,688 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:36,688 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:36,710 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:36,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:36,713 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:43,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:43,161 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:43,161 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:46,723 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:46,723 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:46,740 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:46,742 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:46,743 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:53,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:53,176 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:48:53,176 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:48:56,753 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:48:56,753 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:48:56,779 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:48:56,782 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:48:56,782 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:03,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:03,190 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:03,190 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:06,792 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:06,792 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:06,809 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:06,812 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:06,813 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:13,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:13,214 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:13,216 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:16,825 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:16,825 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:16,854 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:16,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:16,855 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:23,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:23,237 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:23,237 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:26,868 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:26,868 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:26,890 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:26,894 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:26,894 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:33,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:33,260 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:33,260 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:36,904 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:36,904 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:36,920 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:36,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:36,921 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:43,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:43,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:43,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:46,933 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:46,933 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:46,953 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:46,956 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:46,956 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:53,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:53,290 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:49:53,290 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:49:56,966 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:49:56,967 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:49:56,987 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:49:56,990 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:49:56,992 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:03,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:03,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:03,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:07,002 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:07,003 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:07,026 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:07,029 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:07,029 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:13,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:13,328 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:13,328 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:17,039 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:17,039 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:17,059 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:17,062 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:17,062 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:23,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:23,342 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:23,344 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:27,072 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:27,072 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:27,088 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:27,091 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:27,091 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:33,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:33,362 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:33,364 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:37,101 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:37,101 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:37,118 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:37,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:37,119 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:43,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:43,378 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:43,378 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:47,130 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:47,131 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:47,148 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:47,153 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:47,153 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:53,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:53,392 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:50:53,392 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:50:57,163 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:50:57,163 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 77.92 2021-12-16 09:50:57,177 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:50:57,180 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:50:57,180 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:03,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.92 % 2021-12-16 09:51:03,417 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:03,417 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:07,190 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.15 % 2021-12-16 09:51:07,190 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.15 2021-12-16 09:51:07,207 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:07,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:07,210 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:13,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.15 % 2021-12-16 09:51:13,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:13,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:17,220 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.38 % 2021-12-16 09:51:17,220 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.38 2021-12-16 09:51:17,240 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:17,243 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:17,243 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:23,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:23,448 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:23,448 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:27,257 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:27,259 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:51:27,276 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:27,278 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:27,278 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:33,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:33,463 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:33,464 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:37,289 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:37,289 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:51:37,312 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:37,313 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:37,313 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:43,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:43,480 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:43,480 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:47,323 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:47,325 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:51:47,357 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:47,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:47,361 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:53,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:53,494 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:51:53,494 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:51:57,372 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:51:57,372 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:51:57,391 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:51:57,395 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:51:57,395 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:03,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:03,510 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:03,510 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:07,405 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:07,407 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:07,426 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:07,428 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:07,428 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:13,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:13,526 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:13,526 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:17,438 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:17,438 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:17,461 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:17,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:17,463 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:23,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:23,546 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:23,546 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:27,474 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:27,476 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:27,500 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:27,502 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:27,502 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:33,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:33,562 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:33,562 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:37,512 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:37,513 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:37,529 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:37,532 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:37,532 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:43,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:43,578 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:43,578 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:47,543 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:47,543 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:47,569 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:47,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:47,572 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:53,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:53,601 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:52:53,601 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:52:57,588 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:52:57,588 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:52:57,614 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:52:57,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:52:57,617 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:03,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:03,625 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:03,625 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:07,632 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:07,634 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:07,655 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:07,660 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:07,660 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:13,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:13,648 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:13,648 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:17,674 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:17,674 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:17,694 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:17,697 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:17,697 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:23,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:23,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:23,671 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:27,713 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:27,714 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:27,742 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:27,744 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:27,744 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:33,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:33,691 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:33,691 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:37,759 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:37,759 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:37,778 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:37,779 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:37,779 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:43,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:43,710 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:43,710 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:47,799 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:47,801 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:47,828 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:47,832 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:47,832 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:53,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:53,736 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:53:53,736 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:53:57,845 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:53:57,846 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:53:57,867 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:53:57,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:53:57,871 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:03,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:03,765 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:03,765 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:07,888 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:07,888 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:54:07,911 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:07,913 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:07,913 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:13,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:13,785 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:13,785 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:17,927 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:17,927 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:54:17,951 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:17,953 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:17,953 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:23,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:23,805 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:23,805 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:27,969 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.84 % 2021-12-16 09:54:27,970 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 78.84 2021-12-16 09:54:27,996 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:27,996 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:27,997 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:33,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 09:54:33,832 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:33,832 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:38,012 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 09:54:38,012 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.08 2021-12-16 09:54:38,043 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:38,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:38,046 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:43,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.31 % 2021-12-16 09:54:43,855 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:43,855 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:48,061 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:54:48,061 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.77 2021-12-16 09:54:48,078 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:48,081 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:48,081 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:53,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:54:53,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:54:53,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:54:58,095 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:54:58,095 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.77 2021-12-16 09:54:58,119 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:54:58,121 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:54:58,121 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:03,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:03,903 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:55:03,903 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:08,137 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:08,137 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.77 2021-12-16 09:55:08,157 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:55:08,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:55:08,161 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:13,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:13,921 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:55:13,921 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:18,174 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:18,174 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.77 2021-12-16 09:55:18,197 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:55:18,200 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:55:18,200 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:23,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:23,941 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:55:23,941 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:28,223 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:28,223 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 79.77 2021-12-16 09:55:28,249 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:55:28,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:55:28,250 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:33,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.77 % 2021-12-16 09:55:33,969 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:55:33,969 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:38,263 | test | DEBUG | pool-3-thread-7 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.00 % 2021-12-16 09:55:38,263 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: running, progress: 80.0 2021-12-16 09:55:38,285 | infra | DEBUG | pool-3-thread-7 | [task:check:412] Wait before next rebalance progress check 2021-12-16 09:55:38,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 09:55:38,289 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:43,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.00 % 2021-12-16 09:55:43,984 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> rebalance_reached():L111 2021-12-16 09:55:43,984 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 09:55:48,301 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 09:55:48,301 | infra | DEBUG | pool-3-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 09:55:53,312 | test | INFO | pool-3-thread-7 | [task:check:385] Rebalance - status: none, progress: 100 2021-12-16 09:55:53,357 | test | INFO | pool-3-thread-7 | [task:check:444] Rebalance completed with progress: 100% in 8335.75099993 sec 2021-12-16 09:55:53,357 | infra | INFO | pool-3-thread-7 | [task:complete_task:105] Thread 'Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639669017.48' completed 2021-12-16 09:55:53,986 | test | INFO | MainThread | [rest_client:rebalance_reached:121] Rebalance reached >80.0% in 8216.44099998 seconds 2021-12-16 09:55:53,996 | test | DEBUG | MainThread | [rest_client:diag_eval:1461] /diag/eval status on 172.23.110.64:8091: 1 content: false command: ns_orchestrator:needs_rebalance() 2021-12-16 09:55:53,996 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[172.23.110.68]_1639669017.48 2021-12-16 09:55:54,036 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 45859.3, u'diskFetches': 0, u'dataUsed': 10225439848814L, u'itemCount': 4728997231L, u'quotaPercentUsed': 77.68904082849622, u'storageTotals': {u'hdd': {u'usedByData': 12518680803469L, u'used': 12824667509221L, u'quotaTotal': 47967746654208L, u'total': 47967746654208L, u'free': 33184726583876L}, u'ram': {u'usedByData': 333671889616L, u'used': 907154104320L, u'quotaTotal': 776848015360L, u'total': 1079482892288L, u'quotaUsed': 429496729600L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 12518680803469L, u'vbActiveNumNonResident': 4552034255L, u'memUsed': 333671889616L} 2021-12-16 09:55:54,323 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4728997231 | 400.00 GiB | 310.76 GiB | 11.39 TiB | 3.74324890043 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-16 09:55:54,325 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-16 09:55:54,539 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 3.74324890043 2021-12-16 09:55:54,542 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 09:55:54,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:55:54,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 09:55:54,966 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:55:54,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 09:55:54,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 09:55:55,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 09:55:55,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 09:55:55,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 09:55:55,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 09:55:55,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 09:55:55,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 09:55:55,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 09:55:55,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 09:55:55,563 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 09:55:55,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 09:55:56,953 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 09:55:56,956 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 09:55:57,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:55:57,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 09:55:57,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:55:57,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 09:55:57,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 09:55:57,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 09:55:57,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 09:55:57,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 09:55:57,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 09:55:57,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 09:55:57,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 09:55:57,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 09:55:57,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 09:55:58,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 09:55:58,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 09:56:01,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 314 2021-12-16 09:56:01,059 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 353 2021-12-16 09:56:01,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-504 is 321 2021-12-16 09:56:01,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 334 2021-12-16 09:56:01,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 319 2021-12-16 09:56:01,061 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 305 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 329 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-793 is 326 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 314 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 345 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-433 is 330 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-361 is 302 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 325 2021-12-16 09:56:01,062 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 326 2021-12-16 09:56:01,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 320 2021-12-16 09:56:01,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-802 is 327 2021-12-16 09:56:01,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 318 2021-12-16 09:56:01,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-226 is 310 2021-12-16 09:56:01,063 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 333 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 330 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 308 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 326 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 312 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 323 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 322 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 331 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 336 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 326 2021-12-16 09:56:01,065 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 339 2021-12-16 09:56:01,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 308 2021-12-16 09:56:01,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 348 2021-12-16 09:56:01,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 351 2021-12-16 09:56:01,066 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 338 2021-12-16 09:56:01,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 336 2021-12-16 09:56:01,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 338 2021-12-16 09:56:01,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-806 is 306 2021-12-16 09:56:01,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 321 2021-12-16 09:56:01,068 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 371 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-374 is 301 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 339 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 323 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 308 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-447 is 365 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-375 is 306 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-231 is 340 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 321 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 335 2021-12-16 09:56:01,069 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 313 2021-12-16 09:56:01,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 338 2021-12-16 09:56:01,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-809 is 301 2021-12-16 09:56:01,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 334 2021-12-16 09:56:01,071 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-377 is 309 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 336 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 317 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-810 is 307 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 358 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 313 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 331 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 312 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 334 2021-12-16 09:56:01,072 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 317 2021-12-16 09:56:01,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 353 2021-12-16 09:56:01,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 343 2021-12-16 09:56:01,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 322 2021-12-16 09:56:01,073 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 339 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 308 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 310 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-218 is 327 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-74 is 335 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 319 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 317 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 339 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 358 2021-12-16 09:56:01,075 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 318 2021-12-16 09:56:01,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 316 2021-12-16 09:56:01,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 352 2021-12-16 09:56:01,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 332 2021-12-16 09:56:01,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 346 2021-12-16 09:56:01,076 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 313 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 300 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 318 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 322 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-455 is 327 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 334 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 339 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-816 is 310 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 334 2021-12-16 09:56:01,078 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-384 is 311 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 327 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 321 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-817 is 302 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-457 is 327 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-385 is 306 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 320 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 322 2021-12-16 09:56:01,079 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 302 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 339 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-26 is 304 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-819 is 318 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-459 is 334 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-387 is 303 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 369 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 335 2021-12-16 09:56:01,081 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 331 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-820 is 305 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-748 is 303 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 338 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-172 is 302 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-28 is 314 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-821 is 312 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 330 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 328 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 345 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 334 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 328 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 330 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-723 is 305 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 321 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 312 2021-12-16 09:56:01,082 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 350 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-75 is 313 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 328 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 309 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 329 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-390 is 325 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 320 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 339 2021-12-16 09:56:01,084 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 313 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-247 is 324 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 311 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 321 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 336 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 322 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 320 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 386 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 336 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 336 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 309 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-753 is 301 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-465 is 310 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 343 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 353 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 314 2021-12-16 09:56:01,085 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 312 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 329 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 316 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 304 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-827 is 317 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 315 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 342 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 320 2021-12-16 09:56:01,086 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 303 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 316 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 327 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 336 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-396 is 312 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-252 is 328 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 345 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 316 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-469 is 323 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 365 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 344 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-830 is 327 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 335 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 348 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 344 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 336 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-38 is 325 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 324 2021-12-16 09:56:01,088 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 327 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-255 is 307 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 321 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 332 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-796 is 301 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-724 is 306 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-508 is 304 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 310 2021-12-16 09:56:01,089 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 332 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 341 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 317 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 327 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-760 is 321 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 313 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 338 2021-12-16 09:56:01,091 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 306 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 324 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-40 is 302 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-761 is 309 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-473 is 303 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-401 is 313 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 308 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 338 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 321 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 321 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 334 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 357 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 325 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 325 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-403 is 304 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-187 is 305 2021-12-16 09:56:01,092 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 357 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 312 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-692 is 325 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 316 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 320 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 357 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 310 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-837 is 313 2021-12-16 09:56:01,094 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 351 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 325 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 327 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 334 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 310 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 312 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 341 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 307 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 324 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 318 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-191 is 341 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 331 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-696 is 304 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 343 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-408 is 319 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-192 is 328 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-48 is 327 2021-12-16 09:56:01,095 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 323 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-697 is 307 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-481 is 360 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 312 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 354 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 324 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 335 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 311 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 329 2021-12-16 09:56:01,096 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 320 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 336 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 318 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 313 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 334 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 317 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 341 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 350 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 352 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-771 is 335 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 344 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-411 is 320 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 346 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-51 is 361 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 343 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-772 is 318 2021-12-16 09:56:01,098 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-52 is 302 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 315 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-773 is 308 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 329 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-485 is 334 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 313 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 308 2021-12-16 09:56:01,099 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 344 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 310 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-774 is 302 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 333 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-414 is 329 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 339 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 339 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 335 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 333 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 307 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 333 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 337 2021-12-16 09:56:01,101 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 327 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 307 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 315 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 321 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 307 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 346 2021-12-16 09:56:01,102 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 329 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 326 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 310 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 321 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 350 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 312 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 331 2021-12-16 09:56:01,104 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 356 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 348 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 322 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 343 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 349 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 307 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 300 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 319 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-491 is 306 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 328 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 309 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 347 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 309 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-726 is 313 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 326 2021-12-16 09:56:01,105 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 332 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-222 is 339 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 345 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-6 is 341 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-852 is 320 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 324 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 335 2021-12-16 09:56:01,107 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 328 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 348 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 327 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 348 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 302 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 313 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-205 is 311 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-61 is 319 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 324 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 317 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-422 is 331 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 303 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 329 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 318 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 337 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 320 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 303 2021-12-16 09:56:01,108 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-207 is 322 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-63 is 316 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-424 is 302 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 324 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 354 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 330 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 330 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-426 is 313 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 340 2021-12-16 09:56:01,109 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 349 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 312 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 343 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 334 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 317 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 326 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 340 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-428 is 320 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 323 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 332 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 323 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 343 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 338 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-357 is 302 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-213 is 303 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 346 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 333 2021-12-16 09:56:01,111 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 301 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 326 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 343 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 364 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-790 is 330 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 308 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 352 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 339 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 369 2021-12-16 09:56:01,112 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 314 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 355 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 318 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 305 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 326 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 348 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 314 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 354 2021-12-16 09:56:01,114 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 307 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 336 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-729 is 305 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 309 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 308 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 335 2021-12-16 09:56:01,115 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 346 2021-12-16 09:56:01,115 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 09:56:01,118 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 09:56:01,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:56:01,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 09:56:01,519 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:56:01,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 09:56:01,543 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 09:56:01,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 09:56:01,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 09:56:01,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 09:56:01,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 09:56:01,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 09:56:01,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 09:56:02,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 09:56:02,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 09:56:02,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 09:56:02,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 09:56:04,717 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-1008 is 329 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-864 is 310 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-360 is 300 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-144 is 334 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-72 is 323 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-1009 is 303 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 345 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-145 is 326 2021-12-16 09:56:04,719 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-73 is 301 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 347 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-514 is 337 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-1019 is 301 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-875 is 343 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 348 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 328 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-371 is 342 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-948 is 307 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 333 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-372 is 309 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-949 is 309 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-877 is 309 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-517 is 349 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-373 is 319 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-157 is 334 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 316 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-878 is 311 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-518 is 320 2021-12-16 09:56:04,720 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-374 is 307 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 317 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-591 is 312 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-87 is 350 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-88 is 304 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-593 is 354 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-521 is 317 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-377 is 306 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-89 is 301 2021-12-16 09:56:04,721 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-955 is 316 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-595 is 331 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-523 is 316 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 319 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-91 is 305 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-578 is 350 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-362 is 312 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-92 is 302 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-597 is 325 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-93 is 340 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-526 is 332 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-382 is 310 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-959 is 311 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-167 is 313 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-95 is 323 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-960 is 317 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-888 is 319 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-528 is 304 2021-12-16 09:56:04,723 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-168 is 307 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-529 is 330 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-385 is 338 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-98 is 320 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-100 is 318 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-893 is 311 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-101 is 348 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-939 is 301 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-579 is 303 2021-12-16 09:56:04,724 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-363 is 313 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-966 is 327 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-534 is 317 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-391 is 342 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-103 is 334 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-536 is 348 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-104 is 319 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-537 is 301 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-970 is 314 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 338 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-106 is 313 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-107 is 309 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-540 is 318 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-396 is 317 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-108 is 312 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-397 is 335 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-902 is 305 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-542 is 303 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-975 is 305 2021-12-16 09:56:04,726 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-543 is 333 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-399 is 300 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-1012 is 340 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-544 is 350 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-400 is 319 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-112 is 305 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-401 is 326 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-402 is 303 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-547 is 329 2021-12-16 09:56:04,727 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-403 is 320 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-115 is 303 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 314 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 307 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-549 is 302 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 311 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-118 is 305 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-983 is 305 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-551 is 314 2021-12-16 09:56:04,729 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 331 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 328 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-985 is 316 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-121 is 300 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-1013 is 304 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-941 is 335 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-581 is 312 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-149 is 344 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-554 is 315 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-915 is 317 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-555 is 341 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-123 is 319 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 324 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 351 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-989 is 310 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-413 is 320 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-125 is 324 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-342 is 304 2021-12-16 09:56:04,730 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 323 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-343 is 329 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-127 is 302 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-416 is 315 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-993 is 340 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 331 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-345 is 303 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-418 is 326 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-346 is 313 2021-12-16 09:56:04,732 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-130 is 322 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-563 is 308 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 306 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-347 is 330 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-131 is 301 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-1014 is 324 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-582 is 344 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-366 is 348 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-348 is 348 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-132 is 344 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-565 is 338 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-349 is 311 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-133 is 322 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-998 is 324 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-926 is 303 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-566 is 316 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-350 is 335 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-134 is 311 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-567 is 316 2021-12-16 09:56:04,733 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-351 is 328 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-135 is 353 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-928 is 303 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-857 is 317 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-353 is 343 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-137 is 338 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-1002 is 308 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-426 is 331 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-354 is 308 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-138 is 317 2021-12-16 09:56:04,734 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-1003 is 302 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-571 is 305 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-427 is 350 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-355 is 326 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-1005 is 310 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-357 is 316 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-943 is 330 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-367 is 339 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-574 is 349 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-142 is 332 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-1007 is 321 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-944 is 347 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-584 is 308 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-368 is 309 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 319 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-1017 is 351 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-585 is 324 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-513 is 316 2021-12-16 09:56:04,736 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-81 is 317 2021-12-16 09:56:04,737 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 09:56:04,740 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 09:56:04,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:56:05,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 09:56:05,171 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:56:05,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 09:56:05,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 09:56:05,206 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 09:56:05,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 09:56:05,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 09:56:05,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 09:56:05,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 09:56:05,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 09:56:05,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 09:56:05,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 09:56:05,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 09:56:05,788 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 09:56:06,999 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 09:56:07,000 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 11994544MB 2021-12-16 09:56:07,000 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 7644MB 2021-12-16 09:56:07,000 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 414972MB 2021-12-16 09:56:07,000 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 11579241MB 2021-12-16 09:56:07,003 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 09:56:07,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:56:07,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 09:56:07,384 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:56:07,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 09:56:07,405 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 09:56:07,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 09:56:07,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 09:56:07,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 09:56:07,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 09:56:07,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 09:56:07,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 09:56:07,530 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-16 09:56:07,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 09:56:07,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 09:56:08,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 09:56:08,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 09:56:09,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 09:56:09,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 09:56:09,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 09:56:10,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 09:56:10,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 09:56:10,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 09:56:11,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 09:56:11,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 09:56:12,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 09:56:12,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 09:56:13,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 09:56:13,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 09:56:13,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 09:56:14,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 09:56:14,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 09:56:14,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 09:56:15,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 09:56:15,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 09:56:16,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 09:56:16,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 09:56:17,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 09:56:17,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 09:56:17,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 09:56:18,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 09:56:18,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 09:56:18,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 09:56:19,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 09:56:19,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 09:56:20,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 09:56:20,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 09:56:20,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 09:56:21,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 09:56:21,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 09:56:21,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 09:56:22,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 09:56:22,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 09:56:23,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 09:56:23,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 09:56:23,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 09:56:24,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 09:56:24,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 09:56:25,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 09:56:25,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 09:56:25,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 09:56:26,176 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 09:56:26,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 09:56:26,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 09:56:27,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 09:56:27,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 09:56:28,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 09:56:28,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 09:56:28,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 09:56:29,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 09:56:29,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 09:56:29,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 09:56:30,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 09:56:30,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 09:56:31,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 09:56:31,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 09:56:32,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 09:56:32,441 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 09:56:32,444 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 09:56:32,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:56:32,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 09:56:32,923 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:56:32,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 09:56:32,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 09:56:32,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 09:56:32,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 09:56:32,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 09:56:32,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 09:56:33,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 09:56:33,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 09:56:33,068 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.67 - core(s): 72 2021-12-16 09:56:33,068 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 09:56:33,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 09:56:34,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 09:56:34,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 09:56:34,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 09:56:35,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 09:56:35,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 09:56:35,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 09:56:36,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 09:56:36,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 09:56:37,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 09:56:37,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 09:56:37,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 09:56:38,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 09:56:38,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 09:56:39,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 09:56:39,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 09:56:39,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 09:56:40,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 09:56:40,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 09:56:40,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 09:56:41,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 09:56:41,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 09:56:42,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 09:56:42,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 09:56:42,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 09:56:43,255 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 09:56:43,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 09:56:43,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 09:56:44,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 09:56:44,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 09:56:44,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 09:56:45,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 09:56:45,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 09:56:45,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 09:56:46,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 09:56:46,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 09:56:47,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 09:56:47,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 09:56:47,918 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 09:56:48,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 09:56:48,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 09:56:49,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 09:56:49,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 09:56:49,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 09:56:50,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 09:56:50,701 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 09:56:51,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 09:56:51,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 09:56:51,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 09:56:52,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 09:56:52,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 09:56:52,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 09:56:53,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 09:56:53,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 09:56:54,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 09:56:54,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 09:56:54,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 09:56:55,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 09:56:55,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 09:56:55,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 09:56:56,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 09:56:56,655 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 09:56:56,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 09:56:57,391 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 09:56:57,394 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 09:56:57,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:56:57,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 09:56:57,772 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:56:57,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 09:56:57,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 09:56:57,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 09:56:57,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 09:56:57,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 09:56:57,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 09:56:57,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 09:56:57,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 09:56:57,924 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-16 09:56:57,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 09:56:58,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 09:56:58,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 09:56:59,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 09:56:59,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 09:56:59,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 09:57:00,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 09:57:00,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 09:57:01,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 09:57:01,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 09:57:01,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 09:57:02,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 09:57:02,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 09:57:03,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 09:57:03,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 09:57:03,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 09:57:04,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 09:57:04,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 09:57:04,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 09:57:05,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 09:57:05,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 09:57:06,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 09:57:06,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 09:57:06,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 09:57:07,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 09:57:07,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 09:57:07,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 09:57:08,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 09:57:08,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 09:57:09,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 09:57:09,532 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 09:57:09,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 09:57:10,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 09:57:10,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 09:57:11,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 09:57:11,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 09:57:11,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 09:57:12,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 09:57:12,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 09:57:12,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 09:57:13,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 09:57:13,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 09:57:14,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 09:57:14,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 09:57:14,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 09:57:15,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 09:57:15,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 09:57:15,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 09:57:16,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 09:57:16,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 09:57:17,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 09:57:17,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 09:57:17,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 09:57:18,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 09:57:18,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 09:57:19,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 09:57:19,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 09:57:19,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 09:57:20,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 09:57:20,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 09:57:20,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 09:57:21,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 09:57:21,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 09:57:21,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 09:57:22,319 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 09:57:22,322 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 09:57:22,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:57:22,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 09:57:22,793 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:57:22,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 09:57:22,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 09:57:22,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 09:57:22,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 09:57:22,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 09:57:22,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 09:57:22,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 09:57:22,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 09:57:22,993 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-16 09:57:22,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 09:57:23,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 09:57:23,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 09:57:24,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 09:57:24,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 09:57:24,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 09:57:25,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 09:57:25,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 09:57:26,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 09:57:26,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 09:57:26,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 09:57:27,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 09:57:27,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 09:57:28,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 09:57:28,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 09:57:28,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 09:57:29,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 09:57:29,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 09:57:30,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 09:57:30,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 09:57:30,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 09:57:31,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 09:57:31,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 09:57:31,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 09:57:32,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 09:57:32,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 09:57:32,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 09:57:33,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 09:57:33,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 09:57:34,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 09:57:34,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 09:57:34,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 09:57:35,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 09:57:35,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 09:57:35,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 09:57:36,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 09:57:36,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 09:57:37,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 09:57:37,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 09:57:37,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 09:57:38,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 09:57:38,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 09:57:38,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 09:57:39,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 09:57:39,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 09:57:40,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 09:57:40,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 09:57:40,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 09:57:41,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 09:57:41,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 09:57:42,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 09:57:42,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 09:57:42,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 09:57:43,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 09:57:43,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 09:57:44,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 09:57:44,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 09:57:44,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 09:57:45,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 09:57:45,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 09:57:46,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 09:57:46,460 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 09:57:46,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 09:57:47,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 09:57:47,661 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 09:57:47,661 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 105045.796875, u'ReplayWALOffset': u'14109:7045120', u'NTablesCreated': 69282, u'RecentReadIOAmpSet': 0.036018870770931244, u'ReadIOAmp': 0.3893032968044281, u'BlockCacheMisses': 2336701, u'NFlushes': 17478, u'RecentWriteAmp': 1.7634787559509277, u'NumSyncFlushes': 1218, u'TotalMemUsed': 128605180, u'NBloomFilterHits': 32833008, u'Fragmentation': 0.3470339775085449, u'NumNormalFlushes': 8968, u'RecentBlockCacheHitRatio': 78.92402648925781, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 31113, u'ReadIOAmp': 1.6666666666666667, u'MemtablesMemUsed': 3938880, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 425113307, u'CheckpointOverhead': 524288, u'NItemsGet': 3, u'NWriteIOs': 39670, u'NReadBytes': 35645199, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 231596, u'SizeRatio': 0.006902098655700684, u'NWriteBytes': 98648064, u'BloomFilterMemUsed': 0, u'NumTables': 65, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 532480}, {u'TargetSize': 33554432, u'LogicalSize': 25718981, u'SizeRatio': 0.7664853632450104, u'NWriteBytes': 35377152, u'BloomFilterMemUsed': 0, u'NumTables': 16, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 368640}, {u'TargetSize': 533562849, u'LogicalSize': 442148882, u'SizeRatio': 0.8286725412548354, u'NWriteBytes': 5156864, u'BloomFilterMemUsed': 0, u'NumTables': 150, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 26980352}, {u'TargetSize': 27354028210L, u'LogicalSize': 19362340662L, u'SizeRatio': 0.7078423884538357, u'NWriteBytes': 1650688, u'BloomFilterMemUsed': 0, u'NumTables': 912, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 190226432}, {u'TargetSize': 106712570570L, u'LogicalSize': 54708056425L, u'SizeRatio': 0.512667403032085, u'NWriteBytes': 3133784064L, u'BloomFilterMemUsed': 0, u'NumTables': 346, u'WastedSpace': 1351771140, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 54707965952L}], u'NItemsGarbageCollected': 2713626, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 333701579, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 151968, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 47101, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 446406, u'CompactWriter': {u'NDataBlockWrites': 15330, u'NWriteBytes': 44138496, u'NIndexBlockWrites': 1526, u'NBytesIncoming': 58566508, u'NSyncs': 0, u'NInserts': 2662114, u'NWriteIOs': 1500}, u'NWriteBytes': 3449633441L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 5, u'NReadBytes': 12265, u'NDataBlockReads': 3, u'NBytesOutgoing': 0, u'NIndexBlockReads': 6}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 54708056425L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.42479893702100935, u'BufferMemUsed': 365310, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 35345715763L, u'NTableFiles': 1543, u'NFileCountCompacts': 723, u'TableObjectMemUsed': 392584, u'TreeSnapshotMemoryUsed': 178016, u'TotalDiskUsage': 70552137728L, u'BytesIncoming': 8120626349L, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 747, u'NItemsPut': 8337058, u'NSyncs': 8512, u'CompactReader': {u'NReadIOs': 41250, u'NReadBytes': 35179765, u'NDataBlockReads': 27446, u'NBytesOutgoing': 0, u'NIndexBlockReads': 13804}, u'ActiveDiskUsage': 54926073856L, u'NTablesDeleted': 13772, u'IterateReader': {u'NReadIOs': 5846, u'NReadBytes': 453169, u'NDataBlockReads': 2923, u'NBytesOutgoing': 0, u'NIndexBlockReads': 2923}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 977091, u'NWriteBytes': 3405494945L, u'NIndexBlockWrites': 68094, u'NBytesIncoming': 4185456417L, u'NSyncs': 8512, u'NInserts': 2932367, u'NWriteIOs': 38170}}, u'NReadBytes': 6818786483L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 23.766666412353516, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 354737429, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 12610, u'NTxns': 2575750, u'NWriteBytes': 17151217664L, u'NSegments': 1098, u'NSyncs': 2575750, u'NLoggedBytes': 17151217664L, u'NActiveSegments': 1, u'NWrites': 2576135, u'NInserts': 19875320, u'DiskUsed': 14180352}, u'NGets': 1075696, u'NReadBytesSet': 518200142, u'NReadBytesCompact': 5137817746L, u'IndexResidentRatio': 0.053100064396858215, u'CompactQueueSize': 0, u'NSyncsPerSec': 1.600000023841858, u'BlockCacheHitRatio': 40.97697067260742, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 1680515568, u'BloomFilterMemUsedRatio': 0.1935613602399826, u'NWriteBytes': 26233274462L, u'BytesOutgoing': 4226, u'RecentReadIOAmp': 0.5927884578704834, u'WriteAmp': 2.240389108657837, u'NBlocksCached': 27161, u'LogicalDataSize': 36640218962L, u'ActiveBloomFilterMemUsed': 34026656, u'NFileCountCompacts': 2757, u'NTableFiles': 2139, u'FlushQueueSize': 72, u'TableObjectMemUsed': 540392, u'BlockCacheQuota': 95368527, u'TotalDiskUsage': 72112730112L, u'NPurgerRuns': 32, u'ReadIOAmpSet': 0.0, u'NSyncs': 25536, u'NReadIOsSetPerSec': 13.233333587646484, u'NReadIOsPerSec': 48.33333206176758, u'NTablesDeleted': 51987, u'BloomFilterFPR': 0.02204134315252304, u'NBloomFilterHitsPerSec': 1813.9000244140625, u'WriteCacheMemUsed': 5967552, u'RecentBloomFilterFPR': 0.021910274401307106, u'BytesPerRead': 1080.5240478515625, u'BytesIncomingPerSec': 734476.75, u'NReadIOsGetPerSec': 41.099998474121094, u'NGetStats': 14597, u'TotalIndexBlocksSize': 446316343, u'NGetStatsPerSec': 0.8333333134651184, u'NWriteBytesCompact': 5201932288L, u'NReadIOsSet': 201276, u'ActiveDataSize': 36640218962L, u'NWriteBytesCompactPerSec': 4915.2001953125, u'NBloomFilterMissesPerSec': 40.63333511352539, u'NCommitBatches': 2579033, u'NGetStatsComputed': 14597, u'NWriteIOs': 2680054, u'NReadIOsGet': 620048, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 23842132, u'LSMTreeObjectMemUsed': 132608, u'NSetsPerSec': 367.3999938964844, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 1.0666667222976685, u'ManagedFileObjectMemUsed': 209184, u'ReadAmpGet': 275039.125, u'BlockCacheMemUsed': 91306672, u'NWriterCompacts': 0, u'NReadIOs': 2629061, u'WALBufferMemUsed': 65536, u'ReadAmp': 1490910.25, u'keyStats': {u'NItemsTransformed': 115801653, u'NTablesCreated': 19852, u'ReadIOAmp': 0.09288787102501829, u'MemtablesMemUsed': 268864, u'NBloomFilterHits': 32830318, u'TotalIndexBlocksSize': 21194508, u'CheckpointOverhead': 12685312, u'NItemsGet': 6672798, u'NWriteIOs': 37420, u'NReadBytes': 6749993260L, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 1528610, u'SizeRatio': 0.04555612802505493, u'NWriteBytes': 302702592, u'BloomFilterMemUsed': 16440, u'NumTables': 82, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1413120}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 23440333, u'SizeRatio': 0.6985763609409332, u'NWriteBytes': 3199213568L, u'BloomFilterMemUsed': 807680, u'NumTables': 16, u'WastedSpace': 67495, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 23502848}, {u'TargetSize': 140536366, u'LogicalSize': 110956099, u'SizeRatio': 0.789518771248148, u'NWriteBytes': 794091520, u'BloomFilterMemUsed': 3907968, u'NumTables': 57, u'WastedSpace': 287318, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 111235072}, {u'TargetSize': 1405363708, u'LogicalSize': 1405363708, u'SizeRatio': 1.0, u'NWriteBytes': 997261312, u'BloomFilterMemUsed': 29293568, u'NumTables': 191, u'WastedSpace': 4967911, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1410330624}], u'NItemsGarbageCollected': 2958442, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 21033404, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 40512, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 2545163, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 26091, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 128248, u'CompactWriter': {u'NDataBlockWrites': 1867324, u'NWriteBytes': 5149470720L, u'NIndexBlockWrites': 24234, u'NBytesIncoming': 7119372726L, u'NSyncs': 0, u'NInserts': 130190112, u'NWriteIOs': 11609}, u'NWriteBytes': 5477883226L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 619822, u'NReadBytes': 1680292350, u'NDataBlockReads': 9516549, u'NBytesOutgoing': 0, u'NIndexBlockReads': 14504899}, u'NBloomFilterMisses': 739818, u'NItemsMerged': 0, u'LogicalDiskSize': 1405363708, u'TotalBloomFilterMemUsed': 28864696, u'WriteAmp': 19.99018943936432, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 34025656, u'LogicalDataSize': 1294407609, u'NTableFiles': 422, u'NFileCountCompacts': 1018, u'TableObjectMemUsed': 104656, u'TreeSnapshotMemoryUsed': 67504, u'TotalDiskUsage': 1559166976, u'BytesIncoming': 274028580, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 1109, u'NItemsPut': 5592420, u'NSyncs': 8512, u'CompactReader': {u'NReadIOs': 1925341, u'NReadBytes': 5069700910L, u'NDataBlockReads': 1884176, u'NBytesOutgoing': 0, u'NIndexBlockReads': 41165}, u'ActiveDiskUsage': 1546481664, u'NTablesDeleted': 19927, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.022037980424029262, u'Writer': {u'NDataBlockWrites': 56915, u'NWriteBytes': 328412506, u'NIndexBlockWrites': 17301, u'NBytesIncoming': 166832766, u'NSyncs': 8512, u'NInserts': 2876427, u'NWriteIOs': 25811}}, u'NReadBytesCompactPerSec': 5249.2333984375, u'TableMetaMemUsed': 624598, u'NDeletes': 0, u'BlockCacheHitsPerSec': 89.0, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 739994, u'LogicalDiskSize': 56113515723L, u'NSets': 5592420, u'TotalBloomFilterMemUsed': 28865960, u'BufferMemUsed': 365310, u'RecentReadAmp': 0.0, u'NGetsPerSec': 69.33333587646484, u'NInserts': 0, u'NCommitBatchesPerSec': 144.60000610351562, u'FragmentationWithOverhead': 0.4919035732746124, u'NCompactsPerSec': 0.1666666716337204, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 110295.03125, u'TreeSnapshotMemoryUsed': 278048, u'BytesIncoming': 11709249562L, u'NCompacts': 2872, u'NumPersistentFlushes': 7292, u'NWriteIOsPerSec': 150.23333740234375, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 1515.0836181640625, u'ActiveDiskUsage': 56472956928L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.8333333134651184, u'NReadBytesSetPerSec': 33858.5, u'BlockCacheHits': 1622264, u'NWriteBytesPerSec': 1295234.125, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 18317, u'ReadIOAmp': 0.07848011363636363, u'MemtablesMemUsed': 1759808, u'NBloomFilterHits': 2690, u'TotalIndexBlocksSize': 8528, u'CheckpointOverhead': 876544, u'NItemsGet': 2816, u'NWriteIOs': 26829, u'NReadBytes': 33148024, u'LevelStats': [{u'TargetSize': 33554432, u'LogicalSize': 415271, u'SizeRatio': 0.01237604022026062, u'NWriteBytes': 137101312, u'BloomFilterMemUsed': 488, u'NumTables': 41, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 335872}, {u'TargetSize': 33554432, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 33554432, u'LogicalSize': 95590, u'SizeRatio': 0.002848803997039795, u'NWriteBytes': 8060928, u'BloomFilterMemUsed': 512, u'NumTables': 8, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 65536}], u'NItemsGarbageCollected': 6569885, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 2446, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 16704, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 36797, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 49944, u'CompactWriter': {u'NDataBlockWrites': 1016, u'NWriteBytes': 8323072, u'NIndexBlockWrites': 1016, u'NBytesIncoming': 5582155, u'NSyncs': 0, u'NInserts': 53848, u'NWriteIOs': 1016}, u'NWriteBytes': 154540131, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 221, u'NReadBytes': 210953, u'NDataBlockReads': 722, u'NBytesOutgoing': 0, u'NIndexBlockReads': 722}, u'NBloomFilterMisses': 176, u'NItemsMerged': 0, u'LogicalDiskSize': 95590, u'TotalBloomFilterMemUsed': 1264, u'WriteAmp': 0.04186476830545378, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1000, u'LogicalDataSize': 95590, u'NTableFiles': 174, u'NFileCountCompacts': 1016, u'TableObjectMemUsed': 43152, u'TreeSnapshotMemoryUsed': 32528, u'TotalDiskUsage': 1425408, u'BytesIncoming': 3691412547L, u'NItemsDel': 0, u'PendingCheckpointBytes': 155648, u'NCompacts': 1016, u'NItemsPut': 6572815, u'NSyncs': 8512, u'CompactReader': {u'NReadIOs': 36576, u'NReadBytes': 32937071, u'NDataBlockReads': 18288, u'NBytesOutgoing': 0, u'NIndexBlockReads': 18288}, u'ActiveDiskUsage': 401408, u'NTablesDeleted': 18288, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.061409630146545706, u'Writer': {u'NDataBlockWrites': 17301, u'NWriteBytes': 146217059, u'NIndexBlockWrites': 17301, u'NBytesIncoming': 27400241, u'NSyncs': 8512, u'NInserts': 130080, u'NWriteIOs': 25813}}}}} 2021-12-16 09:57:47,665 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.67': [0.004553184844553471, 0.007237884681671858, 0.007537397090345621, 0.0043962262570858, 0.004893206991255283, 0.0048408545553684235, 0.0047808135859668255, 0.004904578439891338, 0.005107628181576729, 0.00494979415088892, 0.004950689617544413, 0.004681719467043877, 0.0048912763595581055, 0.004581431858241558, 0.007076617795974016, 0.0092057129368186, 0.006913293618708849, 0.006825681310147047, 0.007352238986641169, 0.0072188652120530605, 0.006918677594512701, 0.007226203102618456, 0.004939279519021511, 0.0044784690253436565, 0.004782212898135185, 0.004983679857105017, 0.004834630526602268, 0.0048846411518752575, 0.004955230746418238, 0.005085526499897242, 0.004996287636458874, 0.00482936343178153, 0.004931616596877575, 0.005055035464465618, 0.004694024566560984, 0.0050771706737577915, 0.005259758792817593, 0.004926508292555809, 0.005187656730413437, 0.005154982674866915, 0.0053106639534235, 0.0025497619062662125, 0.002584007801488042, 0.002698155352845788, 0.0027436304371804, 0.002429320476949215, 0.0026847338303923607, 0.00279675773344934, 0.002913952339440584, 0.002478821435943246, 0.0027402611449360847, 0.002810921985656023, 0.0024432314094156027, 0.002694579539820552, 0.003046987345442176, 0.002596276579424739, 0.002866602735593915, 0.0024314892943948507, 0.0028352385852485895, 0.0028035491704940796, 0.0028240277897566557, 0.002929799258708954, 0.0027836491353809834, 0.002871979493647814], '172.23.110.65': [0.3470339775085449, 0.273787260055542, 0.2632485330104828, 0.2626197636127472, 0.29903897643089294, 0.31715571880340576, 0.319084107875824, 0.273945152759552, 0.3161362111568451, 0.3252642750740051, 0.3174356520175934, 0.35283300280570984, 0.359457403421402, 0.3538091480731964, 0.31533846259117126, 0.3215937614440918, 0.30846351385116577, 0.31518739461898804, 0.3365969657897949, 0.3198593556880951, 0.29739806056022644, 0.31266433000564575, 0.31111642718315125, 0.349652498960495, 0.33821988105773926, 0.343696266412735, 0.29892978072166443, 0.2684907615184784, 0.28355786204338074, 0.3370809555053711, 0.32399383187294006, 0.3398074805736542, 0.26924771070480347, 0.2815817892551422, 0.2810662090778351, 0.3294927179813385, 0.28431761264801025, 0.2934166491031647, 0.27970409393310547, 0.2960800230503082, 0.3290178179740906, 0.2946677505970001, 0.32558268308639526, 0.2981609106063843, 0.2889498770236969, 0.2759040296077728, 0.2374929040670395, 0.282333642244339, 0.28582778573036194, 0.2866431772708893, 0.2871657907962799, 0.3440355658531189, 0.2737986445426941, 0.2889347970485687, 0.29972490668296814, 0.2904321849346161, 0.3489339053630829, 0.2924133837223053, 0.2915732264518738, 0.3449891209602356, 0.3126160204410553, 0.29247137904167175, 0.3084201514720917, 0.23554591834545135], '172.23.110.66': [0.001960035879164934, 0.0027610198594629765, 0.002737745875492692, 0.002524835988879204, 0.002493175445124507, 0.002566509647294879, 0.0028205551207065582, 0.0025707257445901632, 0.002641878090798855, 0.0026894384063780308, 0.0023682171013206244, 0.002584268106147647, 0.0024570259265601635, 0.002661373233422637, 0.0027445023879408836, 0.0026346994563937187, 0.0025361620355397463, 0.002678615041077137, 0.0024054881650954485, 0.002809845609590411, 0.0028049855027347803, 0.0026306097861379385, 0.0024870934430509806, 0.002706509316340089, 0.0026599918492138386, 0.002458556555211544, 0.002483137883245945, 0.0027203205972909927, 0.002777873305603862, 0.0024768728762865067, 0.002600141568109393, 0.00255030021071434, 0.0026192956138402224, 0.0027320922818034887, 0.0026634843088686466, 0.0025911973789334297, 0.002555845770984888, 0.002741515403613448, 0.0027036580722779036, 0.0023606123868376017, 0.0027608913369476795, 0.0025900958571583033, 0.0025694933719933033, 0.0027664797380566597, 0.0026792557910084724, 0.0024742905516177416, 0.0023536398075520992, 0.0028121506329625845, 0.0027117880526930094, 0.0025990325957536697, 0.002526442985981703, 0.0024096707347780466, 0.0024401673581451178, 0.0025971943978220224, 0.0023888610303401947, 0.002416518284007907, 0.0027317642234265804, 0.002122522098943591, 0.0027503895107656717, 0.0026592998765408993, 0.00259526283480227, 0.0022029748652130365, 0.0027580067981034517, 0.00289378734305501], '172.23.110.64': [0.34899193048477173, 0.3768986463546753, 0.38240835070610046, 0.3669683039188385, 0.3682364523410797, 0.3118801712989807, 0.34618908166885376, 0.3018914759159088, 0.33191728591918945, 0.3257688283920288, 0.33458396792411804, 0.3419821560382843, 0.34261998534202576, 0.30910006165504456, 0.3802255690097809, 0.3665098249912262, 0.32316723465919495, 0.331086128950119, 0.2869254946708679, 0.38353291153907776, 0.32964402437210083, 0.3449559807777405, 0.3567432165145874, 0.32496196031570435, 0.34215468168258667, 0.32994547486305237, 0.2690384089946747, 0.3280630111694336, 0.333278626203537, 0.3714122474193573, 0.3754728138446808, 0.32866376638412476, 0.3458074927330017, 0.3781856596469879, 0.3230564594268799, 0.34367528557777405, 0.3487577736377716, 0.3203430473804474, 0.3850094676017761, 0.3438253104686737, 0.38059720396995544, 0.32975125312805176, 0.30756568908691406, 0.33738476037979126, 0.3457013666629791, 0.2683749794960022, 0.29421567916870117, 0.37504467368125916, 0.33761316537857056, 0.3546231985092163, 0.3843388855457306, 0.37856489419937134, 0.36560800671577454, 0.37027832865715027, 0.3718729019165039, 0.23571743071079254, 0.3935145437717438, 0.3150770664215088, 0.3547859191894531, 0.37767207622528076, 0.27721941471099854, 0.34563305974006653, 0.37139710783958435, 0.3374145030975342]} 2021-12-16 09:57:47,901 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 19.0 2021-12-16 09:57:48,114 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.67 is 19.0 2021-12-16 09:57:48,437 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 19.0 2021-12-16 09:57:48,762 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 19.0 2021-12-16 09:57:48,762 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.67': 19.0, '172.23.110.65': 19.0, '172.23.110.66': 19.0, '172.23.110.64': 19.0} ############################################################ # # Step 10: Crash Magma/memc with Loading of docs # ############################################################ ############################################################ # # RAM FootPrint: 3947.640625 # ############################################################ 2021-12-16 09:57:48,812 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 09:57:48,812 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 100 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-16 09:59:28,816 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 09:59:28,816 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 09:59:33,819 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 09:59:33,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:34,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 09:59:34,197 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:34,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 09:59:34,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 09:59:34,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 09:59:34,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 09:59:34,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 09:59:34,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 09:59:34,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 09:59:34,311 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 09:59:34,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 09:59:34,378 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['225619 memcached\n'], []) 2021-12-16 09:59:34,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 09:59:34,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 09:59:34,525 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['225619 memcached\n'], []) 2021-12-16 09:59:34,526 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 09:59:34,528 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 09:59:34,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:34,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 09:59:34,937 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:34,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 09:59:34,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 09:59:34,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 09:59:34,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 09:59:34,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 09:59:35,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 09:59:35,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 09:59:35,036 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 09:59:35,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 09:59:35,086 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['53047 memcached\n'], []) 2021-12-16 09:59:35,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 09:59:35,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 09:59:35,164 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['53047 memcached\n'], []) 2021-12-16 09:59:35,164 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 09:59:35,167 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 09:59:35,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:35,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 09:59:35,701 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:35,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 09:59:35,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 09:59:35,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 09:59:35,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 09:59:35,778 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 09:59:35,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 09:59:35,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 09:59:35,832 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 09:59:35,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 09:59:35,884 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['132572 memcached\n'], []) 2021-12-16 09:59:35,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 09:59:35,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 09:59:36,000 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['132572 memcached\n'], []) 2021-12-16 09:59:36,000 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 09:59:36,003 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 09:59:36,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:36,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 09:59:36,473 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:36,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 09:59:36,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 09:59:36,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 09:59:36,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 09:59:36,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 09:59:36,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 09:59:36,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 09:59:36,645 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 09:59:36,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 09:59:36,732 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['279919 memcached\n'], []) 2021-12-16 09:59:36,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 09:59:36,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 09:59:36,875 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['279919 memcached\n'], []) 2021-12-16 09:59:36,875 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 09:59:36,878 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 09:59:36,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:37,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 09:59:37,374 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:37,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 09:59:37,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 09:59:37,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 09:59:37,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 09:59:37,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 09:59:37,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 09:59:37,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 09:59:37,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 09:59:37,520 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 09:59:37,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 09:59:37,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:37,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:37,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 09:59:37,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 09:59:37,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:37,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:37,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:37,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:37,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:37,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:37,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:37,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:37,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:37,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:38,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:38,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:38,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:38,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:38,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:38,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:38,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:38,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:38,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:38,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:38,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 09:59:38,282 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 09:59:38,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 09:59:38,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:38,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 09:59:38,328 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 09:59:38,328 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 09:59:38,332 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 09:59:38,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:38,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 09:59:38,782 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:38,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 09:59:38,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 09:59:38,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 09:59:38,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 09:59:38,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 09:59:38,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 09:59:38,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 09:59:38,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 09:59:38,950 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 09:59:38,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 09:59:38,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:39,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:39,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,186 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:39,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:39,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:39,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:39,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:39,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:39,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:39,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:39,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:39,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:39,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:39,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:39,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:39,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:39,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:39,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:39,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:39,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:39,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:39,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:39,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:39,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:40,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:40,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:40,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:40,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:40,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,214 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:40,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:40,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 09:59:40,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 09:59:40,400 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 09:59:40,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 09:59:40,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 09:59:40,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 09:59:40,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 09:59:40,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 09:59:40,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 09:59:40,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 09:59:40,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:40,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 09:59:40,900 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 09:59:40,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 09:59:40,934 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 09:59:40,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 09:59:40,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:40,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 09:59:40,990 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 09:59:40,990 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 09:59:40,993 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 09:59:41,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:41,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 09:59:41,496 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:41,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 09:59:41,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 09:59:41,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 09:59:41,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 09:59:41,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 09:59:41,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 09:59:41,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 09:59:41,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 09:59:41,642 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 09:59:41,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 09:59:41,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:41,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:41,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:41,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:41,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:41,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:41,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:41,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:41,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:42,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:42,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:42,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:42,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:42,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:42,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:42,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:42,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:42,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:42,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:42,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:42,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:42,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:42,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:42,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:42,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:43,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:43,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:43,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:43,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:43,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:43,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:43,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:43,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:43,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:43,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:43,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:43,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 09:59:43,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 09:59:43,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 09:59:43,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 09:59:43,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 09:59:43,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 09:59:43,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 09:59:43,862 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 09:59:43,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:43,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:43,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 09:59:43,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 09:59:43,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 09:59:43,970 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 09:59:43,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 09:59:43,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:44,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 09:59:44,016 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 09:59:44,016 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 09:59:44,019 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 09:59:44,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:44,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 09:59:44,542 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:44,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 09:59:44,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 09:59:44,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 09:59:44,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 09:59:44,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 09:59:44,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 09:59:44,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 09:59:44,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 09:59:44,651 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 09:59:44,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 09:59:44,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:44,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:44,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 09:59:44,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 09:59:44,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:44,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:44,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:44,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 09:59:44,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:45,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 09:59:45,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:45,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 09:59:45,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:45,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 09:59:45,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:45,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 09:59:45,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:45,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 09:59:45,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:45,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 09:59:45,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:45,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 09:59:45,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:45,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 09:59:45,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:45,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 09:59:45,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:45,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 09:59:45,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:45,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 09:59:45,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:45,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:45,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 09:59:45,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:45,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:46,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:46,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 09:59:46,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:46,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 09:59:46,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:46,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 09:59:46,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 09:59:46,118 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 09:59:46,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 09:59:46,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 09:59:46,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 09:59:46,151 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 09:59:46,151 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 09:59:46,151 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 09:59:46,154 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 09:59:46,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 09:59:46,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 09:59:46,661 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 09:59:46,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 09:59:46,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 09:59:46,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 09:59:46,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 09:59:46,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 09:59:46,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 09:59:46,772 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 09:59:46,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:47,500 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-16 09:59:47,507 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:47,507 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 09:59:49,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:49,660 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 09:59:49,664 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:49,664 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 09:59:51,665 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:51,849 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 09:59:51,854 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:51,854 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 09:59:53,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:54,030 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 09:59:54,035 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:54,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 09:59:56,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:56,210 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 09:59:56,216 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:56,216 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 09:59:58,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 09:59:58,381 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 09:59:58,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 09:59:58,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:00,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:00,559 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:00:00,562 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:00,562 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:02,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:02,740 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:00:02,744 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:02,744 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:04,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:04,915 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:00:04,920 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:04,920 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:06,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:07,026 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:00:07,030 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:07,030 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:09,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:09,204 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:00:09,209 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:09,210 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:11,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:11,357 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:11,357 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:13,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:13,509 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:13,509 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:15,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:15,674 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:15,674 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:17,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:17,822 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:17,822 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:19,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:19,976 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:19,976 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:21,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:22,121 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:22,122 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:24,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:24,276 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:24,278 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:26,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:26,417 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:26,417 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:28,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:28,565 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:28,565 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:30,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:30,720 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:30,720 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:32,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:32,892 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:32,892 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:34,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:35,046 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:35,046 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:37,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:37,201 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:37,201 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:39,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:39,362 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:39,362 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:41,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:41,516 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:41,516 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:43,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:43,673 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:43,674 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:45,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:45,828 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:45,828 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:47,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:47,986 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:47,987 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:49,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:50,131 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:50,131 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:52,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:52,282 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:52,282 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:54,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:54,433 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:54,433 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:56,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:56,584 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:56,585 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:00:58,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:00:58,744 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:00:58,744 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:00,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:00,908 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:00,908 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:02,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:03,079 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:03,079 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:05,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:05,239 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:05,239 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:07,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:07,413 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:07,413 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:09,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:09,586 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:09,586 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:11,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:11,763 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:11,763 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:13,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:13,956 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:13,957 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:15,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:16,128 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:16,128 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:18,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:18,434 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:18,434 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:20,436 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:20,592 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:20,592 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:22,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:22,760 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:22,760 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:01:24,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:25,598 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:01:25,599 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 99.4479999542. 2021-12-16 10:01:25,601 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:01:25,602 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:01:25,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:01:26,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:01:26,105 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:01:26,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:01:26,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:01:26,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:01:26,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:01:26,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:01:26,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:01:26,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:01:26,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:26,441 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:01:26,444 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.842000007629. 2021-12-16 10:01:26,444 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:01:26,446 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:01:26,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:01:26,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:01:26,976 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:01:26,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:01:26,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:01:27,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:01:27,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:01:27,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:01:27,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:01:27,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:01:27,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:27,259 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:01:27,259 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.8140001297. 2021-12-16 10:01:27,260 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:01:27,262 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:01:27,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:01:27,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:01:27,878 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:01:27,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:01:27,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:01:27,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:01:27,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:01:27,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:01:27,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:01:27,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:01:27,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:28,164 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:28,165 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:01:30,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:30,335 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:30,335 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:01:32,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:32,506 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:01:32,506 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:01:34,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:01:34,859 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:01:34,861 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.60000014305. ############################################################ # # RAM FootPrint: 3996.7890625 # ############################################################ 2021-12-16 10:01:34,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:01:34,903 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 81 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-16 10:02:55,907 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:02:55,907 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:03:00,910 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:03:01,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:01,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:03:01,674 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:01,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:03:01,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:03:01,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:03:01,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:03:01,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:03:01,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:03:01,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:03:01,897 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:03:01,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:03:02,265 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['250090 memcached\n'], []) 2021-12-16 10:03:02,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:03:02,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:03:02,450 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['250090 memcached\n'], []) 2021-12-16 10:03:02,450 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:03:02,453 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:03:02,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:02,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:03:02,976 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:02,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:03:02,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:03:03,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:03:03,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:03:03,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:03:03,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:03:03,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:03:03,137 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:03:03,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:03:03,229 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['74067 memcached\n'], []) 2021-12-16 10:03:03,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:03:03,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:03:03,375 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['74067 memcached\n'], []) 2021-12-16 10:03:03,375 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:03:03,378 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:03:03,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:03,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:03:03,918 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:03,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:03:03,941 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:03:03,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:03:03,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:03:04,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:03:04,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:03:04,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:03:04,062 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:03:04,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:03:04,151 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['159246 memcached\n'], []) 2021-12-16 10:03:04,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:03:04,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:03:04,295 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['159246 memcached\n'], []) 2021-12-16 10:03:04,295 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:03:04,298 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:03:04,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:04,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:03:04,926 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:04,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:03:04,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:03:04,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:03:05,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:03:05,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:03:05,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:03:05,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:03:05,102 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:03:05,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:03:05,194 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['293026 memcached\n'], []) 2021-12-16 10:03:05,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:03:05,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:03:05,322 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['293026 memcached\n'], []) 2021-12-16 10:03:05,322 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:03:05,325 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:03:05,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:07,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:03:08,068 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:08,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:03:08,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:03:08,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:03:08,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:03:08,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:03:08,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:03:08,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:03:08,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:03:08,230 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:03:08,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:03:08,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:08,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:03:08,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:03:08,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:08,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:08,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:08,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:08,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:08,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:08,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:08,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:08,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:08,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:08,900 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:08,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:08,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:08,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:09,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:09,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:09,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:03:09,059 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:03:09,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:03:09,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:09,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:03:09,105 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:03:09,105 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:03:09,108 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:03:09,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:09,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:03:09,552 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:09,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:03:09,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:03:09,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:03:09,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:03:09,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:03:09,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:03:09,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:03:09,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:03:09,697 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:03:09,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:03:09,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:09,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:09,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:09,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:09,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:09,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:09,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:09,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:09,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:10,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:10,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:10,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:10,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:10,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:10,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:10,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:10,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:10,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:10,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:10,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:10,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:10,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:10,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:10,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:10,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:10,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:10,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:11,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:11,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:11,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:11,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:11,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:11,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:11,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:11,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:11,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:11,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:03:11,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:03:11,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:03:11,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:03:11,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:03:11,716 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:03:11,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:03:11,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:03:11,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:11,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:03:11,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:03:11,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:03:11,928 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:03:11,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:03:11,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:11,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:03:11,976 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:03:11,976 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:03:11,980 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:03:12,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:12,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:03:12,492 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:12,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:03:12,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:03:12,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:03:12,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:03:12,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:03:12,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:03:12,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:03:12,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:03:12,641 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:03:12,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:03:12,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:12,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:12,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:12,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:12,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:12,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:12,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:12,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:12,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:13,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:13,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:13,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:13,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:13,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:13,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:13,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:13,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:13,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:13,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:13,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:13,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:13,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:13,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:13,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:13,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:13,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:13,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:14,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:14,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:14,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:14,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:14,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:14,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:14,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:14,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:14,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:14,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:14,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:14,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:03:14,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:03:14,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:03:14,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:03:14,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:03:14,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:03:14,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:14,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:14,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:03:14,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:03:15,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:03:15,036 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:03:15,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:03:15,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:15,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:03:15,088 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:03:15,088 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:03:15,092 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:03:15,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:15,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:03:15,644 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:15,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:03:15,665 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:03:15,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:03:15,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:03:15,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:03:15,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:03:15,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:03:15,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:03:15,796 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:03:15,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:03:15,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:15,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:15,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:03:15,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:03:16,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:16,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:03:16,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:16,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:03:16,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:16,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:03:16,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:16,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:03:16,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:16,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:03:16,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:16,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:03:16,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:16,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:16,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:03:16,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:16,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:17,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:03:17,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:17,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:03:17,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:17,265 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:03:17,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:17,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:03:17,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:17,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:03:17,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:17,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:03:17,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:17,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:03:17,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:17,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:03:17,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:03:17,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:17,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:03:17,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:03:17,963 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:03:17,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:03:17,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:03:17,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:03:17,999 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:03:17,999 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:03:17,999 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:03:18,002 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:03:18,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:03:18,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:03:18,513 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:03:18,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:03:18,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:03:18,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:03:18,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:03:18,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:03:18,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:03:18,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:03:18,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:18,766 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:18,770 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:18,770 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:20,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:20,959 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:20,963 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:20,963 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:22,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:23,105 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:23,109 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:23,111 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:25,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:25,282 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:25,286 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:25,286 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:27,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:27,434 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:27,440 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:27,440 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:29,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:29,625 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:29,630 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:29,630 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:31,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:31,808 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:03:31,812 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:31,812 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:33,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:33,964 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:33,964 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:35,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:36,125 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:36,125 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:38,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:38,282 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:38,282 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:40,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:40,431 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:40,431 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:42,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:42,573 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:42,573 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:44,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:44,721 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:44,721 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:46,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:46,868 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:46,868 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:48,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:49,009 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:49,009 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:51,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:51,167 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:51,167 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:53,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:53,325 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:53,325 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:55,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:55,482 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:55,482 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:57,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:57,641 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:57,641 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:03:59,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:03:59,793 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:03:59,795 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:01,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:01,950 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:01,950 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:03,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:04,105 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:04,107 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:06,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:06,259 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:06,260 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:08,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:08,411 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:08,413 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:10,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:10,573 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:10,573 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:12,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:12,730 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:12,730 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:14,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:14,890 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:14,891 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:16,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:17,039 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:17,040 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:19,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:19,197 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:19,197 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:21,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:21,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:21,348 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:23,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:23,549 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:23,549 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:25,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:25,775 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:25,775 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:27,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:27,950 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:27,950 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:29,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:30,122 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:30,124 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:32,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:32,642 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:32,644 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:34,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:34,828 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:34,828 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:36,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:37,000 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:37,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:39,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:39,170 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:39,170 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:41,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:41,338 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:41,338 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:43,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:43,500 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:43,500 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:45,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:45,658 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:04:45,660 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:04:47,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:48,003 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:04:48,005 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.006000042. 2021-12-16 10:04:48,006 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:04:48,007 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:04:48,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:04:48,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:04:48,487 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:04:48,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:04:48,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:04:48,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:04:48,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:04:48,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:04:48,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:04:48,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:04:48,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:48,819 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:04:48,819 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.813999891281. 2021-12-16 10:04:48,821 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:04:48,822 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:04:48,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:04:49,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:04:49,344 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:04:49,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:04:49,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:04:49,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:04:49,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:04:49,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:04:49,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:04:49,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:04:49,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:49,625 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:04:49,625 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.804999828339. 2021-12-16 10:04:49,628 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:04:49,628 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:04:50,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:04:51,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:04:51,913 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:04:51,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:04:51,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:04:51,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:04:51,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:04:52,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:04:52,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:04:52,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:04:52,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:04:53,187 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:04:53,187 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.5609998703. ############################################################ # # RAM FootPrint: 3949.90625 # ############################################################ 2021-12-16 10:04:53,226 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:04:53,226 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 114 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-16 10:06:47,232 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:06:47,232 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:06:52,233 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:06:52,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:52,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:06:52,957 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:52,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:06:52,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:06:52,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:06:52,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:06:53,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:06:53,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:06:53,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:06:53,049 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:06:53,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:06:53,107 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['251284 memcached\n'], []) 2021-12-16 10:06:53,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:06:53,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:06:53,203 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['251284 memcached\n'], []) 2021-12-16 10:06:53,203 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:06:53,207 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:06:53,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:53,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:06:53,638 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:53,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:06:53,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:06:53,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:06:53,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:06:53,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:06:53,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:06:53,709 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:06:53,724 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:06:53,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:06:53,770 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['74998 memcached\n'], []) 2021-12-16 10:06:53,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:06:53,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:06:53,885 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['74998 memcached\n'], []) 2021-12-16 10:06:53,887 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:06:53,890 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:06:54,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:54,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:06:54,417 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:54,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:06:54,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:06:54,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:06:54,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:06:54,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:06:54,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:06:54,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:06:54,549 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:06:54,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:06:54,601 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['159887 memcached\n'], []) 2021-12-16 10:06:54,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:06:54,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:06:54,736 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['159887 memcached\n'], []) 2021-12-16 10:06:54,736 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:06:54,740 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:06:55,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:55,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:06:55,380 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:55,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:06:55,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:06:55,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:06:55,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:06:55,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:06:55,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:06:55,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:06:55,526 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:06:55,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:06:55,607 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['294655 memcached\n'], []) 2021-12-16 10:06:55,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:06:55,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:06:55,740 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['294655 memcached\n'], []) 2021-12-16 10:06:55,740 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:06:55,743 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:06:55,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:56,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:06:56,255 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:56,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:06:56,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:06:56,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:06:56,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:06:56,334 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:06:56,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:06:56,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:06:56,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:06:56,401 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:06:56,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:06:56,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:56,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:56,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:06:56,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:06:56,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:56,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:56,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:06:56,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:06:56,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:56,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:56,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:06:56,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:06:56,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:56,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:56,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:06:56,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:06:56,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:57,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:57,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:06:57,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:06:57,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:57,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:57,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:06:57,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:06:57,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:06:57,193 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:06:57,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:06:57,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:57,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:06:57,236 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:06:57,236 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:06:57,240 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:06:57,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:06:57,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:06:57,709 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:06:57,709 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:06:57,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:06:57,747 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:06:57,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:06:57,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:06:57,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:06:57,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:06:57,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:06:57,858 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:06:57,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:06:57,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:57,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:57,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:06:58,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:06:58,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:06:58,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:06:58,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,247 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:06:58,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:06:58,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:06:58,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:06:58,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:06:58,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:06:58,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:06:58,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:06:58,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:06:58,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:06:58,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:58,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:06:58,941 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:06:58,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:58,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:06:59,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:06:59,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:06:59,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:06:59,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:06:59,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:06:59,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:06:59,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:06:59,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:06:59,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:06:59,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:06:59,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:06:59,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:06:59,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:06:59,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:06:59,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:06:59,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:06:59,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:06:59,780 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:06:59,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:06:59,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:06:59,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:06:59,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:07:00,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:07:00,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:00,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:00,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:07:00,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:07:00,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:07:00,184 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:07:00,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:07:00,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:00,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:07:00,239 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:07:00,239 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:07:00,243 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:07:00,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:07:00,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:07:00,773 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:07:00,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:07:00,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:07:00,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:07:00,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:07:00,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:07:00,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:07:00,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:07:00,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:07:00,914 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:07:00,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:07:00,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:07:01,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:07:01,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:07:01,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:07:01,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:07:01,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:07:01,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:07:01,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:07:01,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:07:01,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:07:01,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:07:01,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:07:01,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:07:01,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:07:01,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:07:01,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:07:01,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:07:01,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:07:01,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:07:01,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:07:01,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:01,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:07:01,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:07:01,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:01,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:07:02,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:07:02,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:07:02,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:07:02,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:07:02,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:07:02,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:07:02,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:07:02,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:07:02,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:07:02,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:07:02,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:07:02,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:07:02,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:07:02,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:07:02,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:07:02,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:02,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:07:02,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:07:02,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:07:02,559 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:07:02,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:07:02,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:02,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:07:02,589 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:07:02,589 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:07:02,592 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:07:02,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:07:03,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:07:03,153 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:07:03,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:07:03,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:07:03,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:07:03,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:07:03,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:07:03,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:07:03,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:07:03,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:07:03,259 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:07:03,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:07:03,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:03,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:03,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:07:03,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:07:03,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:03,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:03,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:07:03,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:07:03,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:03,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:03,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:07:03,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:07:03,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:03,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:03,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:07:03,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:07:03,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:03,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:03,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:07:04,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:07:04,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:07:04,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:07:04,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:07:04,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:07:04,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:07:04,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:07:04,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:07:04,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:07:04,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:07:04,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:07:04,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:07:04,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:07:04,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:07:04,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:07:04,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:07:04,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:07:04,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:07:04,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:07:04,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:04,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:04,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:07:05,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:07:05,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:05,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:05,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:07:05,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:07:05,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:05,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:07:05,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:07:05,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:07:05,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:07:05,181 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:07:05,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:07:05,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:07:05,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:07:05,214 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:07:05,214 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:07:05,216 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:07:05,217 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:07:05,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:07:05,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:07:05,706 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:07:05,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:07:05,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:07:05,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:07:05,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:07:05,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:07:05,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:07:05,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:07:05,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:05,986 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:05,990 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:05,990 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:07,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:08,171 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:08,174 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:08,174 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:10,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:10,348 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:10,352 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:10,354 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:12,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:12,532 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:12,536 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:12,536 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:14,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:14,700 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:14,703 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:14,703 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:16,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:16,869 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:16,875 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:16,875 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:18,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:19,042 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:19,046 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:19,048 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:21,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:21,221 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:21,226 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:21,226 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:23,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:23,405 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:07:23,410 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:23,411 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:25,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:25,548 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:25,549 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:27,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:27,691 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:27,691 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:29,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:29,858 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:29,858 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:31,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:32,028 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:32,028 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:34,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:34,176 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:34,176 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:36,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:36,365 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:36,365 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:38,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:38,515 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:38,515 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:40,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:40,658 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:40,658 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:42,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:42,852 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:42,852 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:44,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:45,015 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:45,016 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:47,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:47,174 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:47,174 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:49,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:49,341 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:49,341 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:51,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:51,496 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:51,496 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:53,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:53,654 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:53,654 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:55,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:55,815 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:55,815 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:57,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:07:57,979 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:07:57,979 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:07:59,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:00,131 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:00,131 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:02,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:02,290 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:02,292 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:04,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:04,444 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:04,446 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:06,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:06,604 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:06,604 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:08,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:08,757 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:08,759 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:10,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:10,924 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:10,924 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:12,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:13,078 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:13,078 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:15,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:15,234 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:15,236 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:17,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:17,802 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:17,802 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:19,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:19,964 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:19,964 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:21,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:22,127 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:22,127 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:24,127 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:24,302 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:24,302 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:26,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:26,463 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:26,463 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:28,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:28,631 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:28,631 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:30,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:30,803 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:30,805 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:32,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:32,974 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:32,974 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:34,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:35,144 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:35,145 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:37,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:37,313 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:08:37,315 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:08:39,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:40,944 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:08:40,946 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 95.7289998531. 2021-12-16 10:08:40,947 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:08:40,947 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:08:41,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:08:41,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:08:41,474 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:08:41,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:08:41,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:08:41,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:08:41,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:08:41,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:08:41,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:08:41,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:08:41,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:41,746 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:08:41,746 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.799000024796. 2021-12-16 10:08:41,747 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:08:41,749 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:08:41,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:08:42,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:08:42,279 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:08:42,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:08:42,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:08:42,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:08:42,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:08:42,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:08:42,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:08:42,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:08:42,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:42,618 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:08:42,618 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871000051498. 2021-12-16 10:08:42,619 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:08:42,621 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:08:42,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:08:42,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:08:43,053 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:08:43,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:08:43,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:08:43,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:08:43,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:08:43,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:08:43,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:08:43,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:08:43,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:08:43,361 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:08:43,361 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.741000175476. ############################################################ # # RAM FootPrint: 3911.4375 # ############################################################ 2021-12-16 10:08:43,398 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:08:43,400 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 113 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-16 10:10:36,403 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:10:36,403 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:10:41,405 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:10:41,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:41,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:10:41,838 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:41,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:10:41,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:10:41,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:10:41,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:10:41,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:10:41,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:10:41,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:10:41,924 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:10:41,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:10:41,971 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['253729 memcached\n'], []) 2021-12-16 10:10:41,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:10:42,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:10:42,084 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['253729 memcached\n'], []) 2021-12-16 10:10:42,084 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:10:42,088 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:10:42,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:42,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:10:42,515 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:42,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:10:42,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:10:42,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:10:42,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:10:42,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:10:42,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:10:42,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:10:42,612 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:10:42,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:10:42,693 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['75943 memcached\n'], []) 2021-12-16 10:10:42,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:10:42,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:10:42,795 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['75943 memcached\n'], []) 2021-12-16 10:10:42,795 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:10:42,798 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:10:42,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:43,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:10:43,226 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:43,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:10:43,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:10:43,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:10:43,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:10:43,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:10:43,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:10:43,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:10:43,346 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:10:43,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:10:43,414 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['161164 memcached\n'], []) 2021-12-16 10:10:43,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:10:43,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:10:43,559 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['161164 memcached\n'], []) 2021-12-16 10:10:43,559 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:10:43,561 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:10:43,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:44,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:10:44,134 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:44,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:10:44,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:10:44,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:10:44,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:10:44,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:10:44,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:10:44,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:10:44,302 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:10:44,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:10:44,388 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['296230 memcached\n'], []) 2021-12-16 10:10:44,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:10:44,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:10:44,536 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['296230 memcached\n'], []) 2021-12-16 10:10:44,536 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:10:44,539 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:10:49,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:52,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:10:52,224 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:52,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:10:52,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:10:52,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:10:52,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:10:52,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:10:52,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:10:52,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:10:52,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:10:52,404 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:10:52,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:10:52,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:52,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:52,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:10:52,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:10:52,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:52,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:52,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:10:52,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:10:52,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:52,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:52,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:52,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:52,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:52,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:52,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:52,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:52,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:52,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:52,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:53,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:53,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:53,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:53,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:53,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:53,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:10:53,158 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:10:53,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:10:53,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:53,198 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:10:53,216 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:10:53,216 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:10:53,220 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:10:53,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:53,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:10:53,674 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:53,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:10:53,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:10:53,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:10:53,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:10:53,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:10:53,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:10:53,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:10:53,780 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:10:53,796 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:10:53,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:10:53,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:53,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:53,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:53,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:53,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:54,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:54,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:54,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:54,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:54,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:54,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:10:54,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:10:54,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:10:54,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:10:54,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:10:54,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:10:54,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:10:54,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:10:54,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:54,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:10:54,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:10:54,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:54,984 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:10:55,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:10:55,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:10:55,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:10:55,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:10:55,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:10:55,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:10:55,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:10:55,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:10:55,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:10:55,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:10:55,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:10:55,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:10:55,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:10:55,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:55,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:55,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:10:55,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:10:55,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:56,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:56,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:10:56,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:10:56,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:56,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:56,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:10:56,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:10:56,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:56,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:56,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:10:56,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:10:56,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:10:56,453 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:10:56,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:10:56,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:56,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:10:56,516 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:10:56,516 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:10:56,520 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:10:56,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:10:56,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:10:57,059 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:10:57,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:10:57,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:10:57,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:10:57,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:10:57,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:10:57,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:10:57,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:10:57,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:10:57,211 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:10:57,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:10:57,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:57,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:57,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:10:57,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:57,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:57,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:10:57,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:57,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:57,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:10:57,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:57,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:57,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:10:57,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:57,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:10:57,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:10:57,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:57,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:10:58,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:10:58,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:10:58,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:10:58,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:10:58,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:10:58,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:10:58,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:10:58,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:10:58,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:10:58,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:10:58,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:10:58,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:10:58,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:10:58,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:58,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:58,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:10:58,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:10:58,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:10:59,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:10:59,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:10:59,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:10:59,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:10:59,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:10:59,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:10:59,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:10:59,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:10:59,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:10:59,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:10:59,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:10:59,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:10:59,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:10:59,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:10:59,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:10:59,801 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:10:59,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:10:59,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:10:59,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:10:59,845 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:10:59,845 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:10:59,848 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:10:59,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:11:00,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:11:00,239 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:11:00,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:11:00,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:11:00,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:11:00,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:11:00,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:11:00,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:11:00,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:11:00,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:11:00,351 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:11:00,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:11:00,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:11:00,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:11:00,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:11:00,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:11:00,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:11:00,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:11:00,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:11:00,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:11:00,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:11:00,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:11:00,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:00,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:11:00,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:11:00,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:00,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:11:01,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:11:01,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:11:01,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:11:01,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:11:01,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:11:01,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:11:01,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:11:01,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:11:01,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:11:01,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:11:01,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:11:01,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:01,900 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:11:01,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:11:01,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:01,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:02,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:11:02,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:11:02,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:02,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:02,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:11:02,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:11:02,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:02,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:02,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:11:02,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:11:02,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:02,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:11:02,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:11:02,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:11:02,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:11:02,808 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:11:02,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:11:02,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:11:02,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:11:02,937 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:11:02,938 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:11:02,938 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:11:02,941 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:11:03,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:11:03,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:11:03,476 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:11:03,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:11:03,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:11:03,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:11:03,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:11:03,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:11:03,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:11:03,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:11:03,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:03,766 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:11:03,770 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:03,770 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:05,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:05,943 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:11:05,947 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:05,947 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:07,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:08,125 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:11:08,130 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:08,130 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:10,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:10,312 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:11:10,319 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:10,319 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:12,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:12,486 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:11:12,490 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:12,492 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:14,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:14,727 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:14,727 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:16,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:16,874 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:16,875 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:18,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:19,023 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:19,025 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:21,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:21,168 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:21,168 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:23,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:23,321 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:23,321 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:25,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:25,479 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:25,479 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:27,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:27,618 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:27,618 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:29,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:29,766 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:29,766 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:31,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:31,915 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:31,915 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:33,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:34,142 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:34,142 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:36,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:36,302 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:36,302 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:38,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:38,464 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:38,464 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:40,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:40,635 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:40,635 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:42,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:42,795 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:42,796 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:44,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:44,954 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:44,956 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:46,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:47,111 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:47,111 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:49,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:49,273 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:49,273 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:51,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:51,430 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:51,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:53,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:53,588 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:53,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:55,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:55,743 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:55,744 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:57,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:11:57,904 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:11:57,905 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:11:59,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:00,065 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:00,065 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:02,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:02,226 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:02,226 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:04,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:04,384 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:04,384 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:06,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:06,545 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:06,545 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:08,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:08,785 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:08,786 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:10,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:10,963 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:10,963 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:12,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:13,730 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:13,730 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:15,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:15,979 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:15,979 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:17,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:18,140 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:18,140 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:20,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:20,469 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:20,469 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:22,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:22,663 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:22,663 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:24,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:24,944 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:24,944 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:26,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:27,114 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:12:27,114 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:12:29,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:29,434 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:12:29,434 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 86.4960000515. 2021-12-16 10:12:29,436 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:12:29,438 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:12:29,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:12:29,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:12:29,946 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:12:29,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:12:29,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:12:29,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:12:30,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:12:30,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:12:30,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:12:30,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:12:30,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:30,259 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:12:30,259 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.822999954224. 2021-12-16 10:12:30,259 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:12:30,262 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:12:30,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:12:30,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:12:30,792 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:12:30,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:12:30,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:12:30,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:12:30,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:12:30,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:12:30,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:12:30,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:12:30,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:31,076 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:12:31,076 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.817999839783. 2021-12-16 10:12:31,076 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:12:31,078 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:12:31,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:12:31,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:12:31,463 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:12:31,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:12:31,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:12:31,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:12:31,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:12:31,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:12:31,533 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:12:31,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:12:31,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:12:31,727 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:12:31,729 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.652000188828. ############################################################ # # RAM FootPrint: 3857.92578125 # ############################################################ 2021-12-16 10:12:31,760 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:12:31,760 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 106 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-16 10:14:17,765 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:14:17,765 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:14:22,766 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:14:23,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:24,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:14:24,424 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:24,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:14:24,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:14:24,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:14:24,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:14:24,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:14:24,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:14:24,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:14:24,568 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:14:24,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:14:24,661 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['254921 memcached\n'], []) 2021-12-16 10:14:24,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:14:24,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:14:24,770 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['254921 memcached\n'], []) 2021-12-16 10:14:24,770 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:14:24,773 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:14:24,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:25,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:14:25,279 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:25,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:14:25,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:14:25,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:14:25,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:14:25,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:14:25,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:14:25,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:14:25,438 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:14:25,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:14:25,520 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['76663 memcached\n'], []) 2021-12-16 10:14:25,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:14:25,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:14:25,624 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['76663 memcached\n'], []) 2021-12-16 10:14:25,625 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:14:25,628 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:14:25,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:26,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:14:26,167 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:26,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:14:26,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:14:26,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:14:26,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:14:26,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:14:26,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:14:26,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:14:26,299 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:14:26,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:14:26,367 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['162420 memcached\n'], []) 2021-12-16 10:14:26,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:14:26,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:14:26,467 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['162420 memcached\n'], []) 2021-12-16 10:14:26,467 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:14:26,470 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:14:26,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:26,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:14:26,967 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:26,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:14:26,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:14:27,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:14:27,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:14:27,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:14:27,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:14:27,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:14:27,144 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:14:27,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:14:27,229 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['297112 memcached\n'], []) 2021-12-16 10:14:27,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:14:27,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:14:27,371 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['297112 memcached\n'], []) 2021-12-16 10:14:27,371 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:14:27,375 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:14:27,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:27,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:14:28,025 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:28,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:14:28,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:14:28,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:14:28,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:14:28,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:14:28,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:14:28,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:14:28,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:14:28,165 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:14:28,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:14:28,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:14:28,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:14:28,339 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:14:28,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:14:28,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:28,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:28,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:28,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:28,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:28,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:28,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:28,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:28,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:28,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:28,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:14:28,990 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:14:28,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:14:29,007 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:29,028 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:14:29,049 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:14:29,051 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:14:29,052 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:14:29,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:29,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:14:29,513 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:29,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:14:29,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:14:29,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:14:29,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:14:29,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:14:29,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:14:29,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:14:29,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:14:29,684 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:14:29,709 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:14:29,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:29,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:29,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:29,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:29,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:29,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:29,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:29,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:29,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:30,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:30,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:30,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:30,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:30,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:30,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:30,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:30,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:30,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:30,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:30,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:30,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:30,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:30,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:30,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:30,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:30,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:30,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:30,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:30,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:30,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:30,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:30,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:30,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:31,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:31,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:14:31,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:14:31,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:14:31,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:14:31,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:14:31,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:14:31,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:14:31,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:14:31,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:14:31,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:14:31,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:31,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:14:31,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:14:31,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:14:31,516 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:14:31,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:14:31,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:31,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:14:31,556 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:14:31,556 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:14:31,559 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:14:31,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:31,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:14:32,062 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:32,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:14:32,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:14:32,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:14:32,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:14:32,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:14:32,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:14:32,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:14:32,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:14:32,193 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:14:32,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:14:32,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:32,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:32,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:32,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:32,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:32,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:32,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:32,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:32,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:32,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:32,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:32,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:32,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:32,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:32,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:32,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:32,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:33,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:33,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:33,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:33,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:33,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:33,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:33,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:33,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:33,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:33,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:33,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:33,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:33,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:33,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:33,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:33,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:33,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:33,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:14:33,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:14:33,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:34,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:34,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:14:34,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:14:34,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:34,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:34,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:14:34,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:14:34,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:34,247 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:34,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:14:34,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:14:34,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:34,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:34,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:14:34,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:14:34,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:14:34,418 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:14:34,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:14:34,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:34,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:14:34,516 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:14:34,516 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:14:34,519 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:14:34,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:34,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:14:35,032 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:35,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:14:35,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:14:35,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:14:35,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:14:35,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:14:35,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:14:35,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:14:35,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:14:35,196 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:14:35,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:14:35,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:14:35,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:14:35,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:14:35,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:14:35,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:35,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:14:35,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:35,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:14:35,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:35,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:14:35,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:35,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:35,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:14:35,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:35,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:36,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:14:36,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:36,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:14:36,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:36,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:14:36,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:36,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:14:36,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:36,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:14:36,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:36,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:14:36,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:36,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:14:36,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:36,747 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:14:36,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:36,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:14:36,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:36,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:36,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:14:36,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:36,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:14:37,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:37,038 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:14:37,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:14:37,073 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:14:37,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:14:37,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:14:37,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:14:37,115 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:14:37,115 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:14:37,115 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:14:37,118 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:14:37,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:14:37,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:14:37,609 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:14:37,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:14:37,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:14:37,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:14:37,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:14:37,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:14:37,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:14:37,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:14:37,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:37,924 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:37,931 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:37,931 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:39,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:40,085 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:40,089 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:40,089 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:42,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:42,188 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:42,193 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:42,193 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:44,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:44,335 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:44,341 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:44,341 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:46,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:46,529 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:46,535 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:46,535 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:48,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:48,707 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:48,711 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:48,711 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:50,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:50,884 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:50,888 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:50,888 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:52,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:53,059 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:14:53,062 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:53,062 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:55,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:55,272 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:55,272 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:57,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:57,418 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:57,418 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:14:59,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:14:59,563 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:14:59,565 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:01,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:01,743 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:01,743 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:03,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:03,904 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:03,904 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:05,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:06,043 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:06,045 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:08,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:08,200 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:08,200 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:10,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:10,346 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:10,346 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:12,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:12,519 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:12,519 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:14,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:14,663 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:14,663 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:16,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:16,822 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:16,822 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:18,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:18,963 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:18,963 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:20,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:21,115 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:21,117 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:23,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:23,276 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:23,276 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:25,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:25,434 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:25,434 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:27,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:27,594 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:27,595 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:29,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:29,759 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:29,759 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:31,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:31,914 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:31,915 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:33,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:34,072 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:34,073 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:36,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:36,219 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:36,220 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:38,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:38,369 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:38,369 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:40,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:40,522 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:40,523 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:42,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:42,681 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:42,681 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:44,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:44,845 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:44,845 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:46,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:47,163 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:47,164 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:49,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:49,447 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:49,448 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:51,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:51,839 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:51,839 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:53,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:54,030 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:54,030 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:56,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:56,203 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:56,203 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:15:58,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:15:58,380 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:15:58,380 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:16:00,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:00,627 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:16:00,627 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:16:02,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:02,789 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:16:02,789 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:16:04,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:04,969 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:16:04,969 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:16:06,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:07,418 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:16:07,418 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 90.3029999733. 2021-12-16 10:16:07,418 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:16:07,423 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:16:07,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:16:07,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:16:07,937 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:16:07,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:16:07,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:16:07,977 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:16:08,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:16:08,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:16:08,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:16:08,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:16:08,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:08,302 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:16:08,302 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.883000135422. 2021-12-16 10:16:08,302 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:16:08,305 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:16:08,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:16:08,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:16:08,839 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:16:08,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:16:08,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:16:08,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:16:08,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:16:08,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:16:08,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:16:08,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:16:08,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:09,098 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:16:09,098 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.796000003815. 2021-12-16 10:16:09,101 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:16:09,101 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:16:09,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:16:09,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:16:09,540 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:16:09,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:16:09,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:16:09,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:16:09,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:16:09,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:16:09,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:16:09,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:16:09,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:16:09,950 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:16:09,950 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.850000143051. ############################################################ # # RAM FootPrint: 3812.48828125 # ############################################################ 2021-12-16 10:16:09,983 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:16:09,983 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 73 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-16 10:17:22,989 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:17:22,989 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:17:27,990 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:17:28,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:29,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:17:29,505 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:29,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:17:29,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:17:29,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:17:29,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:17:29,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:17:29,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:17:29,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:17:29,713 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:17:29,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:17:29,766 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['257406 memcached\n'], []) 2021-12-16 10:17:29,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:17:29,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:17:29,963 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['257406 memcached\n'], []) 2021-12-16 10:17:29,963 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:17:29,967 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:17:30,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:30,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:17:30,470 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:30,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:17:30,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:17:30,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:17:30,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:17:30,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:17:30,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:17:30,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:17:30,634 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:17:30,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:17:30,726 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['77828 memcached\n'], []) 2021-12-16 10:17:30,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:17:30,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:17:30,877 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['77828 memcached\n'], []) 2021-12-16 10:17:30,877 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:17:30,880 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:17:30,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:31,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:17:31,405 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:31,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:17:31,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:17:31,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:17:31,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:17:31,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:17:31,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:17:31,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:17:31,569 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:17:31,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:17:31,654 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['163663 memcached\n'], []) 2021-12-16 10:17:31,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:17:31,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:17:31,796 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['163663 memcached\n'], []) 2021-12-16 10:17:31,796 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:17:31,799 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:17:31,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:32,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:17:32,273 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:32,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:17:32,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:17:32,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:17:32,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:17:32,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:17:32,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:17:32,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:17:32,434 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:17:32,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:17:32,519 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['299431 memcached\n'], []) 2021-12-16 10:17:32,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:17:32,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:17:32,638 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['299431 memcached\n'], []) 2021-12-16 10:17:32,638 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:17:32,641 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:17:32,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:33,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:17:33,171 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:33,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:17:33,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:17:33,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:17:33,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:17:33,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:17:33,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:17:33,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:17:33,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:17:33,334 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:17:33,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:17:33,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:17:33,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:17:33,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:17:33,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:17:33,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:33,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:33,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:33,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:33,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:33,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:33,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:33,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:33,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:33,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:33,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:33,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:33,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:17:34,012 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:17:34,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:17:34,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:34,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:17:34,046 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:17:34,048 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:17:34,051 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:17:34,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:34,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:17:34,500 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:34,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:17:34,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:17:34,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:17:34,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:17:34,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:17:34,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:17:34,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:17:34,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:17:34,654 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:17:34,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:17:34,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:34,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:34,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:34,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:34,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:34,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:34,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:34,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:34,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:34,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:34,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:35,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:35,028 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:35,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:35,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:35,198 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:35,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:35,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:35,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:35,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:35,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:35,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:35,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:35,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:35,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:35,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:35,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:35,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:35,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:35,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:35,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:35,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:35,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:35,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:35,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:36,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:36,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:17:36,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:17:36,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:17:36,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:17:36,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:17:36,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:17:36,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:17:36,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:17:36,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:17:36,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:17:36,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:36,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:17:36,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:17:36,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:17:36,552 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:17:36,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:17:36,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:36,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:17:36,601 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:17:36,601 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:17:36,605 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:17:36,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:37,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:17:37,114 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:37,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:17:37,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:17:37,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:17:37,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:17:37,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:17:37,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:17:37,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:17:37,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:17:37,256 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:17:37,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:17:37,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:37,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:37,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:37,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:37,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:37,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:37,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:37,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:37,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:37,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:37,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:37,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:37,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:37,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:38,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:38,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:38,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:38,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:38,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:38,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:38,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:38,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:38,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:38,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:38,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,668 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:38,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:38,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:38,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:38,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:38,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:38,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:38,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:38,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:38,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:38,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:17:38,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:17:39,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:39,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:39,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:17:39,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:17:39,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:39,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:39,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:17:39,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:17:39,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:39,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:39,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:17:39,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:17:39,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:39,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:39,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:17:39,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:17:39,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:17:39,263 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:17:39,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:17:39,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:39,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:17:39,296 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:17:39,296 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:17:39,299 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:17:39,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:39,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:17:39,799 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:39,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:17:39,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:17:39,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:17:39,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:17:39,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:17:39,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:17:39,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:17:39,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:17:39,944 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:17:39,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:17:39,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:17:40,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:17:40,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:17:40,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:17:40,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,334 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:40,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:17:40,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,533 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:40,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:17:40,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:40,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:17:40,716 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:40,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:17:40,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:40,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:17:40,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:40,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:40,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:40,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:17:40,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:41,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:17:41,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:41,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:17:41,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:41,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:17:41,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:41,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:17:41,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:41,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:17:41,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:41,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:17:41,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:41,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:17:41,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:41,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:41,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:17:41,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:41,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:17:42,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:42,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:17:42,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:17:42,069 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:17:42,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:17:42,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:17:42,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:17:42,117 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:17:42,118 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:17:42,118 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:17:42,119 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:17:42,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:17:42,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:17:42,634 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:17:42,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:17:42,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:17:42,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:17:42,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:17:42,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:17:42,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:17:42,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:17:42,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:42,963 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:42,967 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:42,967 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:44,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:45,109 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:45,115 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:45,115 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:47,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:47,298 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:47,301 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:47,302 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:49,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:49,470 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:49,474 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:49,474 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:51,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:51,598 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:51,601 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:51,601 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:53,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:53,775 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:53,779 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:53,779 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:55,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:55,957 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:55,960 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:55,961 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:17:57,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:17:58,128 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:17:58,134 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:17:58,134 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:00,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:00,285 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:00,285 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:02,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:02,428 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:02,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:04,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:04,578 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:04,578 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:06,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:06,778 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:06,778 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:08,778 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:08,926 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:08,926 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:10,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:11,072 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:11,072 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:13,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:13,210 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:13,210 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:15,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:15,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:15,348 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:17,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:17,506 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:17,507 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:19,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:19,658 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:19,658 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:21,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:21,828 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:21,828 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:23,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:23,986 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:23,986 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:25,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:26,141 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:26,141 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:28,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:28,295 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:28,296 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:30,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:30,454 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:30,454 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:32,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:32,612 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:32,614 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:34,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:34,772 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:34,772 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:36,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:36,930 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:36,930 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:38,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:39,096 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:39,096 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:41,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:41,240 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:41,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:43,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:43,398 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:43,398 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:45,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:45,546 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:45,548 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:47,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:47,697 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:47,697 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:49,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:49,855 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:49,855 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:51,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:52,127 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:52,127 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:54,127 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:54,342 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:54,342 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:56,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:56,555 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:56,555 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:18:58,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:18:58,732 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:18:58,733 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:00,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:01,009 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:19:01,009 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:03,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:03,184 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:19:03,184 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:05,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:05,444 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:19:05,446 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:07,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:07,617 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:19:07,617 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:09,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:09,789 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:19:09,789 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:19:11,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:12,051 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:19:12,051 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 89.9330000877. 2021-12-16 10:19:12,052 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:19:12,053 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:19:12,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:19:12,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:19:12,556 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:19:12,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:19:12,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:19:12,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:19:12,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:19:12,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:19:12,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:19:12,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:19:12,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:12,917 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:19:12,917 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.865999937057. 2021-12-16 10:19:12,917 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:19:12,921 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:19:13,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:19:13,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:19:13,426 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:19:13,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:19:13,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:19:13,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:19:13,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:19:13,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:19:13,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:19:13,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:19:13,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:13,753 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:19:13,753 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.835999965668. 2021-12-16 10:19:13,755 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:19:13,756 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:19:14,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:19:14,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:19:14,742 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:19:14,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:19:14,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:19:14,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:19:14,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:19:14,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:19:14,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:19:14,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:19:14,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:19:15,073 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:19:15,073 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.3189997673. ############################################################ # # RAM FootPrint: 3770.91015625 # ############################################################ 2021-12-16 10:19:15,105 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:19:15,107 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 74 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-16 10:20:29,109 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:20:29,109 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:20:34,111 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:20:34,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:35,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:20:35,539 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:35,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:20:35,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:20:35,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:20:35,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:20:35,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:20:35,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:20:35,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:20:35,726 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:20:35,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:20:35,803 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['259215 memcached\n'], []) 2021-12-16 10:20:35,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:20:36,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:20:36,275 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['259215 memcached\n'], []) 2021-12-16 10:20:36,275 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:20:36,278 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:20:36,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:36,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:20:36,782 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:36,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:20:36,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:20:36,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:20:36,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:20:36,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:20:36,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:20:36,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:20:36,940 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:20:36,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:20:37,020 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['78747 memcached\n'], []) 2021-12-16 10:20:37,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:20:37,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:20:37,127 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['78747 memcached\n'], []) 2021-12-16 10:20:37,127 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:20:37,131 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:20:37,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:37,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:20:37,650 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:37,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:20:37,668 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:20:37,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:20:37,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:20:37,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:20:37,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:20:37,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:20:37,809 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:20:37,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:20:37,877 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['164877 memcached\n'], []) 2021-12-16 10:20:37,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:20:37,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:20:37,987 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['164877 memcached\n'], []) 2021-12-16 10:20:37,987 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:20:37,992 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:20:38,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:38,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:20:38,480 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:38,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:20:38,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:20:38,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:20:38,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:20:38,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:20:38,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:20:38,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:20:38,632 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:20:38,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:20:38,698 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['300986 memcached\n'], []) 2021-12-16 10:20:38,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:20:38,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:20:38,852 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['300986 memcached\n'], []) 2021-12-16 10:20:38,852 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:20:38,855 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:20:38,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:39,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:20:40,017 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:40,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:20:40,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:20:40,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:20:40,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:20:40,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:20:40,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:20:40,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:20:40,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:20:40,138 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:20:40,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:20:40,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:20:40,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:20:40,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:20:40,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:20:40,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:40,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:40,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:40,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:40,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:40,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:40,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:40,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:40,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:40,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:40,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:40,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:20:40,872 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:20:40,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:20:40,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:40,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:20:40,918 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:20:40,918 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:20:40,921 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:20:41,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:41,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:20:41,397 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:41,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:20:41,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:20:41,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:20:41,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:20:41,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:20:41,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:20:41,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:20:41,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:20:41,558 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:20:41,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:20:41,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:41,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:41,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:41,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:41,716 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:41,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:41,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:41,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:41,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:41,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:41,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:41,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:41,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:41,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:41,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:42,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:42,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:42,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:42,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:42,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:42,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:42,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:42,339 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:42,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:42,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:42,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:42,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:42,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:42,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:42,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:42,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:42,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:42,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:42,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:42,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:42,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:43,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:43,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:43,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:20:43,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:20:43,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:20:43,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:20:43,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:20:43,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:20:43,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:20:43,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:20:43,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:20:43,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:20:43,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:43,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:20:43,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:20:43,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:20:43,579 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:20:43,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:20:43,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:43,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:20:43,615 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:20:43,615 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:20:43,618 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:20:43,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:44,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:20:44,131 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:44,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:20:44,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:20:44,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:20:44,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:20:44,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:20:44,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:20:44,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:20:44,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:20:44,285 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:20:44,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:20:44,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:44,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:44,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:44,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:44,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:44,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:44,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:44,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:44,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:44,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:44,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:44,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:44,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:44,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:44,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:44,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:44,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:44,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:44,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:44,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:45,028 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:45,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:45,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:45,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:45,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:45,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:45,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:45,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:45,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:45,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:45,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:45,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:45,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:45,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:45,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:45,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:46,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:46,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:46,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:46,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:46,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:46,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:46,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:20:46,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:20:46,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:20:46,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:20:46,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:46,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:20:46,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:20:46,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:46,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:47,028 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:20:47,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:20:47,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:47,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:47,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:20:47,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:20:47,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:20:47,227 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:20:47,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:20:47,247 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:47,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:20:47,283 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:20:47,283 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:20:47,288 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:20:47,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:47,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:20:47,826 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:47,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:20:47,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:20:47,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:20:47,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:20:47,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:20:47,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:20:47,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:20:47,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:20:47,990 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:20:48,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:20:48,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:20:48,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:20:48,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:20:48,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:20:48,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:48,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:20:48,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:48,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:20:48,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:48,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:20:48,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:48,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:20:48,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:48,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:48,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:48,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:20:49,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:49,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:20:49,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:49,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:20:49,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,318 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:49,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:20:49,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:49,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:20:49,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:49,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:20:49,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:49,716 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:20:49,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:49,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:20:49,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:49,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:49,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:50,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:20:50,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:50,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:50,130 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:50,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:20:50,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:50,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:20:50,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:50,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:20:50,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:20:50,315 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:20:50,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:20:50,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:20:50,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:20:50,355 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:20:50,357 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:20:50,358 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:20:50,361 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:20:50,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:20:50,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:20:50,891 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:20:50,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:20:50,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:20:50,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:20:50,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:20:50,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:20:51,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:20:51,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:20:51,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:20:51,194 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:20:51,197 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:20:51,197 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:20:53,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:20:53,384 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:20:53,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:20:53,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:20:55,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:20:55,535 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:20:55,536 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:20:57,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:20:57,697 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:20:57,697 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:20:59,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:20:59,839 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:20:59,839 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:01,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:02,003 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:02,003 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:04,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:04,157 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:04,157 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:06,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:06,301 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:06,301 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:08,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:08,444 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:08,444 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:10,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:10,615 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:10,615 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:12,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:12,765 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:12,765 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:14,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:14,915 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:14,915 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:16,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:17,072 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:17,073 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:19,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:19,236 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:19,236 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:21,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:21,397 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:21,397 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:23,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:23,552 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:23,552 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:25,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:25,710 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:25,710 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:27,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:27,865 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:27,865 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:29,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:30,026 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:30,026 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:32,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:32,180 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:32,181 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:34,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:34,339 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:34,339 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:36,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:36,499 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:36,499 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:38,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:38,655 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:38,655 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:40,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:40,813 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:40,815 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:42,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:42,963 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:42,963 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:44,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:45,121 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:45,121 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:47,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:47,276 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:47,276 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:49,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:49,430 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:49,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:51,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:51,569 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:51,569 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:53,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:53,714 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:53,714 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:55,716 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:55,875 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:55,875 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:21:57,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:21:58,036 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:21:58,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:22:00,038 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:00,184 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:00,184 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:22:02,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:02,361 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:02,361 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:22:04,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:04,676 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:04,677 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:22:06,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:07,948 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:22:07,948 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 77.5909998417. 2021-12-16 10:22:07,950 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:22:07,951 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:22:08,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:22:08,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:22:08,454 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:22:08,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:22:08,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:22:08,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:22:08,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:22:08,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:22:08,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:22:08,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:22:08,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:08,813 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:22:08,815 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.865000009537. 2021-12-16 10:22:08,815 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:22:08,818 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:22:08,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:22:09,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:22:09,342 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:22:09,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:22:09,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:22:09,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:22:09,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:22:09,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:22:09,436 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:22:09,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:22:09,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:09,665 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:22:09,667 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.851000070572. 2021-12-16 10:22:09,667 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:22:09,670 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:22:09,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:22:09,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:22:09,996 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:22:09,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:22:10,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:22:10,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:22:10,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:22:10,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:22:10,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:22:10,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:22:10,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:10,262 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:10,263 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:22:12,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:12,427 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:12,427 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:22:14,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:14,588 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:22:14,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:22:16,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:22:16,992 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:22:16,993 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.32500004768. ############################################################ # # RAM FootPrint: 3752.41015625 # ############################################################ 2021-12-16 10:22:17,026 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:22:17,026 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 95 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-16 10:23:52,029 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:23:52,029 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:23:57,032 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:23:57,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:23:57,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:23:57,953 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:23:57,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:23:57,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:23:57,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:23:58,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:23:58,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:23:58,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:23:58,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:23:58,062 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:23:58,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:23:58,124 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['260788 memcached\n'], []) 2021-12-16 10:23:58,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:23:58,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:23:58,838 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['260788 memcached\n'], []) 2021-12-16 10:23:58,838 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:23:58,841 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:23:58,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:23:59,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:23:59,352 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:23:59,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:23:59,372 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:23:59,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:23:59,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:23:59,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:23:59,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:23:59,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:23:59,494 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:23:59,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:23:59,562 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['79671 memcached\n'], []) 2021-12-16 10:23:59,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:23:59,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:23:59,644 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['79671 memcached\n'], []) 2021-12-16 10:23:59,644 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:23:59,647 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:23:59,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:00,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:24:00,167 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:00,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:24:00,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:24:00,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:24:00,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:24:00,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:24:00,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:24:00,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:24:00,305 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:24:00,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:24:00,378 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['166084 memcached\n'], []) 2021-12-16 10:24:00,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:24:00,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:24:00,492 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['166084 memcached\n'], []) 2021-12-16 10:24:00,492 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:24:00,494 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:24:00,637 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:00,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:24:01,029 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:01,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:24:01,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:24:01,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:24:01,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:24:01,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:24:01,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:24:01,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:24:01,209 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:24:01,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:24:01,299 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['302536 memcached\n'], []) 2021-12-16 10:24:01,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:24:01,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:24:01,431 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['302536 memcached\n'], []) 2021-12-16 10:24:01,433 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:24:01,434 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:24:01,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:02,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:24:02,392 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:02,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:24:02,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:24:02,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:24:02,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:24:02,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:24:02,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:24:02,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:24:02,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:24:02,539 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:24:02,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:24:02,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:02,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:02,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:24:02,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:24:02,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:02,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:02,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:24:02,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:24:02,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:02,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:02,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:02,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:02,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:02,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:02,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:03,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:03,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:03,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:03,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:03,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:03,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:03,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:03,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:03,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:03,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:03,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:03,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:03,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:03,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:24:03,384 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:24:03,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:24:03,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:03,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:24:03,436 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:24:03,436 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:24:03,438 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:24:03,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:03,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:24:03,904 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:03,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:24:03,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:24:03,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:24:03,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:24:03,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:24:04,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:24:04,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:24:04,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:24:04,059 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:24:04,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:24:04,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:04,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:04,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,318 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:04,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:04,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:04,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:04,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:04,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:04,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:04,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:04,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:04,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:04,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:04,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:04,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,900 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:04,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:04,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:04,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:04,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:05,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:05,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:05,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:05,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:05,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:05,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:05,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:05,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:05,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:05,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:05,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:05,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:05,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:05,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:24:05,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:24:05,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:24:05,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:24:05,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:05,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:05,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:24:05,984 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:24:06,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:06,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:06,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:24:06,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:24:06,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:06,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:06,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:24:06,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:24:06,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:24:06,177 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:24:06,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:24:06,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:06,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:24:06,219 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:24:06,219 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:24:06,221 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:24:06,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:06,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:24:06,726 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:06,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:24:06,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:24:06,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:24:06,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:24:06,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:24:06,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:24:06,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:24:06,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:24:06,862 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:24:06,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:24:06,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:06,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:06,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:07,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:07,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:07,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:07,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:07,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:07,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:07,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:07,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:07,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:07,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:07,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:07,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:07,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:07,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:07,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:07,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:08,007 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:08,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:08,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:08,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:08,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:08,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:08,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:08,372 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:08,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:08,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:08,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:08,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:08,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:08,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:08,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:08,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:08,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:08,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:09,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:09,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:09,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:09,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:09,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:24:09,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:24:09,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:09,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:09,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:24:09,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:24:09,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:09,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:09,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:24:09,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:24:09,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:09,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:09,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:24:09,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:24:09,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:24:09,642 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:24:09,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:24:09,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:09,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:24:09,694 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:24:09,694 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:24:09,697 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:24:09,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:10,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:24:10,191 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:10,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:24:10,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:24:10,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:24:10,247 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:24:10,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:24:10,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:24:10,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:24:10,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:24:10,367 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:24:10,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:24:10,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:10,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:10,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:24:10,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:24:10,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:10,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:10,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:24:10,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:24:10,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:10,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:10,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:10,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:24:10,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:10,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:10,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:11,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:24:11,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:11,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:24:11,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:11,265 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:24:11,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:11,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:24:11,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:11,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:24:11,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:11,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:24:11,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:11,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:24:11,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:11,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:11,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:11,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:24:11,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:12,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:24:12,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:12,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:24:12,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:12,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:24:12,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:12,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:24:12,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:12,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:24:12,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:12,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:24:12,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:12,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:24:12,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:12,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:24:12,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:24:12,990 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:24:12,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:24:13,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:24:13,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:24:13,051 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:24:13,051 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:24:13,051 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:24:13,053 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:24:13,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:24:13,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:24:13,565 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:24:13,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:24:13,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:24:13,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:24:13,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:24:13,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:24:13,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:24:13,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:24:13,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:13,871 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:13,875 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:13,875 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:15,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:16,062 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:16,065 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:16,065 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:18,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:18,246 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:18,250 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:18,250 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:20,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:20,430 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:20,434 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:20,434 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:22,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:22,602 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:22,607 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:22,607 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:24,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:24,778 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:24,782 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:24,782 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:26,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:26,950 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:26,953 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:26,953 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:28,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:29,130 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:24:29,134 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:29,134 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:31,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:31,273 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:31,275 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:33,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:33,418 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:33,420 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:35,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:35,562 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:35,562 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:37,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:37,716 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:37,716 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:39,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:39,878 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:39,878 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:41,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:42,036 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:42,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:44,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:44,177 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:44,177 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:46,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:46,322 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:46,322 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:48,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:48,494 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:48,496 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:50,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:50,640 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:50,640 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:52,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:52,792 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:52,793 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:54,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:54,948 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:54,950 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:56,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:57,111 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:57,111 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:24:59,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:24:59,267 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:24:59,267 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:01,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:01,421 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:01,421 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:03,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:03,578 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:03,578 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:05,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:05,727 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:05,727 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:07,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:07,884 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:07,885 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:09,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:10,039 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:10,039 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:12,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:12,211 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:12,213 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:14,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:14,371 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:14,371 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:16,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:16,536 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:16,538 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:18,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:18,694 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:18,694 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:20,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:21,200 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:21,200 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:23,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:23,390 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:23,390 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:25,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:25,601 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:25,601 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:27,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:27,882 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:27,882 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:29,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:30,045 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:30,045 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:32,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:32,240 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:32,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:34,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:34,434 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:34,434 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:36,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:36,618 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:36,619 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:38,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:38,799 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:38,799 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:40,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:40,969 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:40,970 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:42,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:43,171 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:25:43,171 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:25:45,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:45,628 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:25:45,628 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 92.5779998302. 2021-12-16 10:25:45,628 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:25:45,631 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:25:45,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:25:46,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:25:46,161 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:25:46,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:25:46,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:25:46,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:25:46,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:25:46,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:25:46,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:25:46,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:25:46,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:46,533 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:25:46,535 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.906000137329. 2021-12-16 10:25:46,535 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:25:46,536 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:25:46,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:25:46,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:25:47,038 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:25:47,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:25:47,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:25:47,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:25:47,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:25:47,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:25:47,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:25:47,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:25:47,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:47,257 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:25:47,259 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.723999977112. 2021-12-16 10:25:47,259 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:25:47,262 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:25:47,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:25:47,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:25:47,674 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:25:47,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:25:47,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:25:47,709 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:25:47,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:25:47,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:25:47,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:25:47,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:25:47,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:25:47,987 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:25:47,989 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.729000091553. ############################################################ # # RAM FootPrint: 3704.53515625 # ############################################################ 2021-12-16 10:25:48,019 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:25:48,019 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-16 10:27:31,025 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:27:31,026 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:27:36,028 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:27:36,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:36,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:27:36,690 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:36,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:27:36,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:27:36,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:27:36,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:27:36,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:27:36,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:27:36,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:27:36,793 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:27:36,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:27:36,854 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['262559 memcached\n'], []) 2021-12-16 10:27:36,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:27:36,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:27:36,941 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['262559 memcached\n'], []) 2021-12-16 10:27:36,941 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:27:36,946 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:27:37,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:37,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:27:37,328 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:37,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:27:37,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:27:37,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:27:37,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:27:37,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:27:37,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:27:37,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:27:37,456 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:27:37,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:27:37,513 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['80620 memcached\n'], []) 2021-12-16 10:27:37,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:27:37,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:27:37,614 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['80620 memcached\n'], []) 2021-12-16 10:27:37,614 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:27:37,618 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:27:37,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:37,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:27:38,046 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:38,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:27:38,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:27:38,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:27:38,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:27:38,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:27:38,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:27:38,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:27:38,160 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:27:38,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:27:38,224 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['167327 memcached\n'], []) 2021-12-16 10:27:38,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:27:38,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:27:38,342 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['167327 memcached\n'], []) 2021-12-16 10:27:38,342 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:27:38,344 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:27:38,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:38,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:27:38,790 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:38,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:27:38,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:27:38,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:27:38,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:27:38,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:27:38,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:27:38,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:27:38,930 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:27:38,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:27:38,983 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['1471 memcached\n'], []) 2021-12-16 10:27:38,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:27:39,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:27:39,131 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['1471 memcached\n'], []) 2021-12-16 10:27:39,131 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:27:39,134 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:27:39,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:40,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:27:40,283 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:40,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:27:40,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:27:40,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:27:40,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:27:40,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:27:40,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:27:40,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:27:40,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:27:40,436 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:27:40,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:27:40,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:40,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:40,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:27:40,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:27:40,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:40,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:40,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:27:40,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:27:40,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:40,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:40,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:40,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:40,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:40,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:40,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:40,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:40,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:41,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:41,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:41,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:41,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:41,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:41,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:41,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:41,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:41,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:41,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:41,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:41,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:27:41,374 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:27:41,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:27:41,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:41,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:27:41,424 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:27:41,426 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:27:41,427 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:27:41,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:41,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:27:41,872 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:41,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:27:41,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:27:41,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:27:41,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:27:41,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:27:41,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:27:41,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:27:42,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:27:42,022 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:27:42,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:27:42,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:42,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:42,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:42,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:42,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:42,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:42,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:42,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:42,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:42,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:42,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:42,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:42,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:42,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:42,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:42,780 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:42,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:42,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:42,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:42,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:42,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:42,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:42,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:43,043 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:43,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:43,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:43,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:43,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:43,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:43,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:43,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:43,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:43,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:27:43,515 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:27:43,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:27:43,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:27:43,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:27:43,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:27:43,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:27:43,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:27:43,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:43,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:27:43,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:27:43,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:27:43,826 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:27:43,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:27:43,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:43,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:27:43,859 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:27:43,859 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:27:43,862 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:27:43,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:44,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:27:44,365 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:44,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:27:44,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:27:44,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:27:44,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:27:44,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:27:44,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:27:44,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:27:44,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:27:44,525 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:27:44,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:27:44,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:44,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:44,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:44,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:44,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:44,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:44,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:44,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:44,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:44,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:44,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:44,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:45,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:45,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:45,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:45,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:45,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:45,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:45,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:45,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:45,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:45,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:45,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:45,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:45,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:45,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:45,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:46,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:46,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:46,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:46,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:46,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:46,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:46,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:46,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:46,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:46,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:46,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:46,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:46,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:46,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:46,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:46,977 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:27:47,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:27:47,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:47,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:47,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:27:47,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:27:47,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:47,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:47,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:27:47,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:27:47,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:47,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:47,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:27:47,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:27:47,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:27:47,421 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:27:47,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:27:47,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:47,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:27:47,470 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:27:47,470 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:27:47,473 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:27:47,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:47,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:27:47,986 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:47,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:27:48,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:27:48,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:27:48,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:27:48,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:27:48,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:27:48,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:27:48,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:27:48,181 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:27:48,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:27:48,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:27:48,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:27:48,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:27:48,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:27:48,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:48,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:27:48,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:48,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:27:48,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:48,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:27:48,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:48,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:48,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:48,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:27:48,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:49,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:27:49,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:49,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:27:49,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:49,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:27:49,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:49,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:27:49,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:49,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:27:49,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:49,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:27:49,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:49,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:27:49,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:49,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:49,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:50,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:27:50,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:50,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:50,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:50,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:27:50,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:50,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:50,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:50,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:27:50,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:50,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:50,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:50,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:27:50,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:50,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:27:50,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:50,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:27:50,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:27:50,592 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:27:50,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:27:50,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:27:50,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:27:50,654 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:27:50,654 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:27:50,654 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:27:50,657 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:27:50,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:27:51,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:27:51,165 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:27:51,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:27:51,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:27:51,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:27:51,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:27:51,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:27:51,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:27:51,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:27:51,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:27:51,437 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:27:51,443 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:27:51,443 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:27:53,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:27:53,619 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:27:53,625 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:27:53,625 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:27:55,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:27:55,802 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:27:55,805 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:27:55,805 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:27:57,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:27:57,982 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:27:57,986 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:27:57,986 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:27:59,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:00,155 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:28:00,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:00,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:02,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:02,341 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:28:02,344 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:02,345 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:04,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:04,516 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:28:04,519 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:04,519 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:06,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:06,688 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:28:06,693 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:06,693 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:08,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:08,864 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:28:08,869 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:08,869 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:10,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:11,023 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:11,023 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:13,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:13,170 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:13,171 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:15,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:15,316 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:15,318 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:17,318 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:17,479 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:17,479 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:19,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:19,627 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:19,627 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:21,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:21,782 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:21,782 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:23,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:23,946 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:23,946 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:25,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:26,092 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:26,092 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:28,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:28,236 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:28,236 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:30,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:30,440 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:30,440 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:32,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:32,608 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:32,608 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:34,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:34,769 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:34,769 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:36,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:36,927 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:36,927 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:38,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:39,078 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:39,078 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:41,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:41,239 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:41,239 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:43,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:43,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:43,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:45,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:45,536 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:45,536 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:47,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:47,700 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:47,700 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:49,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:49,855 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:49,855 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:51,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:52,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:52,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:54,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:54,157 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:54,158 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:56,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:56,308 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:56,309 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:28:58,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:28:58,463 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:28:58,463 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:00,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:00,614 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:00,614 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:02,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:02,766 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:02,766 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:04,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:04,924 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:04,924 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:06,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:07,081 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:07,082 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:09,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:09,387 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:09,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:11,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:11,555 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:11,555 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:13,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:13,726 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:13,726 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:15,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:15,898 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:15,898 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:17,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:18,078 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:18,079 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:20,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:20,243 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:20,243 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:22,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:22,404 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:29:22,404 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:29:24,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:24,579 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:29:24,579 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 93.9249999523. 2021-12-16 10:29:24,581 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:29:24,584 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:29:24,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:29:25,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:29:25,105 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:29:25,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:29:25,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:29:25,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:29:25,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:29:25,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:29:25,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:29:25,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:29:25,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:25,434 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:29:25,434 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.853999853134. 2021-12-16 10:29:25,436 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:29:25,437 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:29:25,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:29:25,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:29:25,943 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:29:25,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:29:25,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:29:25,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:29:26,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:29:26,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:29:26,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:29:26,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:29:26,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:26,227 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:29:26,227 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.791999816895. 2021-12-16 10:29:26,230 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:29:26,230 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:29:26,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:29:27,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:29:27,263 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:29:27,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:29:27,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:29:27,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:29:27,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:29:27,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:29:27,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:29:27,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:29:27,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:29:27,549 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:29:27,551 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.32100009918. ############################################################ # # RAM FootPrint: 3876.51171875 # ############################################################ 2021-12-16 10:29:27,582 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:29:27,584 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-16 10:31:19,589 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:31:19,589 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:31:24,592 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:31:24,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:24,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:31:25,016 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:25,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:31:25,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:31:25,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:31:25,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:31:25,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:31:25,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:31:25,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:31:25,114 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:31:25,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:31:25,161 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['264570 memcached\n'], []) 2021-12-16 10:31:25,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:31:25,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:31:25,275 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['264570 memcached\n'], []) 2021-12-16 10:31:25,275 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:31:25,278 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:31:25,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:25,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:31:25,681 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:25,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:31:25,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:31:25,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:31:25,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:31:25,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:31:25,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:31:25,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:31:25,835 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:31:25,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:31:25,917 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['81577 memcached\n'], []) 2021-12-16 10:31:25,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:31:25,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:31:26,033 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['81577 memcached\n'], []) 2021-12-16 10:31:26,033 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:31:26,038 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:31:26,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:26,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:31:26,552 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:26,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:31:26,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:31:26,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:31:26,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:31:26,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:31:26,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:31:26,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:31:26,677 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:31:26,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:31:26,759 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['168599 memcached\n'], []) 2021-12-16 10:31:26,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:31:26,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:31:26,910 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['168599 memcached\n'], []) 2021-12-16 10:31:26,910 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:31:26,913 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:31:27,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:27,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:31:27,407 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:27,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:31:27,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:31:27,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:31:27,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:31:27,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:31:27,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:31:27,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:31:27,584 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:31:27,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:31:27,673 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['3130 memcached\n'], []) 2021-12-16 10:31:27,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:31:27,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:31:27,816 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['3130 memcached\n'], []) 2021-12-16 10:31:27,816 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:31:27,819 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:31:27,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:35,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:31:35,701 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:35,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:31:35,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:31:35,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:31:35,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:31:35,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:31:35,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:31:35,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:31:35,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:31:35,878 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:31:35,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:31:35,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:35,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:31:36,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:31:36,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:31:36,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:31:36,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:31:36,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:31:36,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:36,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:36,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:36,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:36,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:36,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:36,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:36,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:36,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:36,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:31:36,897 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:31:36,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:31:36,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:36,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:31:36,944 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:31:36,946 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:31:36,947 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:31:37,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:37,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:31:37,493 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:37,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:31:37,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:31:37,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:31:37,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:31:37,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:31:37,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:31:37,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:31:37,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:31:37,686 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:31:37,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:31:37,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:37,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:37,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:37,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:37,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:37,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:37,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:38,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:38,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:38,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:38,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:38,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:38,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:38,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:38,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:38,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:38,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:38,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:38,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:38,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:38,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:38,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:38,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:38,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:38,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:38,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:39,007 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:39,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:39,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:39,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:39,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,334 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:39,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:39,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:39,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:39,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:39,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:39,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:31:39,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:31:39,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:31:39,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:31:39,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:39,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:39,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:31:40,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:31:40,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:40,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:40,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:31:40,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:31:40,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:40,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:40,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:31:40,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:31:40,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:31:40,282 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:31:40,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:31:40,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:40,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:31:40,326 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:31:40,326 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:31:40,328 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:31:40,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:40,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:31:40,815 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:40,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:31:40,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:31:40,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:31:40,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:31:40,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:31:40,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:31:40,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:31:40,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:31:40,959 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:31:40,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:31:41,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:41,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:41,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:41,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:41,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:41,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:41,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:41,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:41,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:41,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:41,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:41,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:41,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:41,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:41,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:41,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:41,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:41,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:41,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:41,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:41,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:42,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:42,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:42,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:42,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:42,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:42,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:42,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:42,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:42,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:42,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:42,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:42,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:31:42,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:31:42,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:31:42,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:31:42,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:42,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:42,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:31:43,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:31:43,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:43,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:43,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:31:43,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:31:43,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:43,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:43,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:31:43,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:31:43,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:31:43,187 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:31:43,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:31:43,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:43,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:31:43,229 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:31:43,230 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:31:43,232 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:31:43,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:43,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:31:43,762 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:43,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:31:43,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:31:43,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:31:43,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:31:43,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:31:43,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:31:43,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:31:43,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:31:43,897 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:31:43,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:31:43,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:43,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:31:44,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:31:44,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:31:44,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:31:44,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:31:44,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:31:44,334 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:44,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:31:44,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:44,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:31:44,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:44,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:31:44,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:44,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:31:44,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:44,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:31:44,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:44,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:44,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:44,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:31:44,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:45,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:31:45,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:45,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:31:45,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,214 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:45,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:31:45,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:45,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:31:45,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:45,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:31:45,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:45,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:31:45,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:45,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:31:45,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:45,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:31:45,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:31:45,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:45,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:31:45,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:31:45,937 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:31:45,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:31:45,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:31:45,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:31:45,983 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:31:45,983 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:31:45,983 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:31:45,986 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:31:46,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:31:46,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:31:46,505 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:31:46,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:31:46,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:31:46,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:31:46,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:31:46,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:31:46,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:31:46,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:31:46,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:46,765 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:46,769 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:46,769 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:48,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:48,933 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:48,937 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:48,937 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:50,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:51,115 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:51,118 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:51,118 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:53,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:53,299 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:53,303 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:53,303 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:55,303 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:55,477 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:55,482 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:55,482 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:57,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:57,665 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:57,670 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:57,671 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:31:59,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:31:59,801 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:31:59,805 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:31:59,805 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:01,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:01,982 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:32:01,984 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:01,986 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:03,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:04,157 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:32:04,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:04,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:06,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:06,301 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:06,301 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:08,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:08,457 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:08,457 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:10,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:10,612 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:10,614 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:12,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:12,756 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:12,756 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:14,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:14,903 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:14,903 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:16,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:17,053 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:17,055 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:19,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:19,214 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:19,214 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:21,214 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:21,371 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:21,371 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:23,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:23,526 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:23,526 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:25,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:25,670 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:25,670 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:27,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:27,839 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:27,839 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:29,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:30,005 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:30,005 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:32,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:32,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:32,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:34,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:34,328 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:34,328 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:36,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:36,490 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:36,492 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:38,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:38,651 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:38,651 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:40,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:40,805 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:40,805 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:42,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:42,954 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:42,954 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:44,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:45,114 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:45,114 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:47,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:47,286 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:47,286 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:49,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:49,443 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:49,443 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:51,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:51,598 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:51,599 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:53,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:53,757 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:53,757 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:55,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:55,918 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:55,920 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:32:57,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:32:58,078 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:32:58,078 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:00,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:00,247 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:00,247 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:02,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:02,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:02,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:04,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:04,546 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:04,548 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:06,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:06,704 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:06,704 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:08,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:08,862 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:08,864 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:10,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:11,023 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:11,025 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:13,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:13,178 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:13,178 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:15,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:15,392 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:15,392 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:17,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:17,588 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:33:17,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:33:19,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:19,770 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:33:19,772 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 93.7890000343. 2021-12-16 10:33:19,772 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:33:19,773 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:33:19,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:33:20,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:33:20,263 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:33:20,265 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:33:20,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:33:20,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:33:20,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:33:20,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:33:20,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:33:20,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:33:20,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:20,602 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:33:20,602 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.830999851227. 2021-12-16 10:33:20,604 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:33:20,605 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:33:20,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:33:21,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:33:21,099 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:33:21,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:33:21,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:33:21,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:33:21,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:33:21,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:33:21,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:33:21,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:33:21,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:21,398 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:33:21,398 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.793999910355. 2021-12-16 10:33:21,398 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:33:21,400 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:33:21,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:33:21,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:33:21,986 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:33:21,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:33:22,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:33:22,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:33:22,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:33:22,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:33:22,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:33:22,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:33:22,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:33:22,306 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:33:22,306 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.907999992371. ############################################################ # # RAM FootPrint: 4166.26953125 # ############################################################ 2021-12-16 10:33:22,336 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:33:22,336 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 86 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-16 10:34:48,341 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:34:48,341 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:34:53,342 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:34:53,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:54,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:34:54,112 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:54,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:34:54,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:34:54,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:34:54,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:34:54,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:34:54,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:34:54,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:34:54,305 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:34:54,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:34:54,371 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['265755 memcached\n'], []) 2021-12-16 10:34:54,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:34:54,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:34:54,632 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['265755 memcached\n'], []) 2021-12-16 10:34:54,634 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:34:54,635 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:34:54,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:55,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:34:55,131 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:55,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:34:55,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:34:55,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:34:55,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:34:55,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:34:55,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:34:55,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:34:55,282 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:34:55,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:34:55,346 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['82268 memcached\n'], []) 2021-12-16 10:34:55,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:34:55,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:34:55,434 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['82268 memcached\n'], []) 2021-12-16 10:34:55,436 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:34:55,438 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:34:55,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:55,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:34:55,947 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:55,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:34:55,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:34:55,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:34:55,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:34:56,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:34:56,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:34:56,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:34:56,068 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:34:56,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:34:56,148 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['169902 memcached\n'], []) 2021-12-16 10:34:56,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:34:56,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:34:56,273 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['169902 memcached\n'], []) 2021-12-16 10:34:56,273 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:34:56,276 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:34:56,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:56,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:34:56,753 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:56,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:34:56,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:34:56,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:34:56,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:34:56,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:34:56,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:34:56,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:34:56,934 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:34:56,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:34:57,029 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['3920 memcached\n'], []) 2021-12-16 10:34:57,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:34:57,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:34:57,187 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['3920 memcached\n'], []) 2021-12-16 10:34:57,187 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:34:57,190 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:34:57,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:57,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:34:57,684 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:57,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:34:57,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:34:57,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:34:57,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:34:57,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:34:57,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:34:57,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:34:57,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:34:57,838 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:34:57,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:34:57,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:57,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:57,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:34:57,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:34:58,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:34:58,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:34:58,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:34:58,198 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:34:58,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:34:58,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:34:58,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:34:58,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:34:58,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:34:58,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:34:58,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:58,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:34:58,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:34:58,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:34:58,575 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:34:58,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:34:58,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:58,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:34:58,635 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:34:58,635 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:34:58,638 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:34:58,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:34:59,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:34:59,094 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:34:59,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:34:59,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:34:59,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:34:59,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:34:59,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:34:59,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:34:59,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:34:59,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:34:59,239 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:34:59,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:34:59,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:34:59,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:34:59,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:34:59,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:34:59,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:34:59,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:34:59,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:34:59,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:34:59,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:34:59,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:34:59,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,778 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:34:59,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:34:59,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:34:59,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:34:59,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:34:59,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:34:59,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:00,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:00,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:00,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:00,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:00,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:00,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:00,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:00,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:00,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:00,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:00,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:00,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:00,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:00,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:00,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:00,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:35:00,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:35:00,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:35:00,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:35:00,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:35:00,862 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:35:00,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:00,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:35:00,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:35:00,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:00,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:01,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:35:01,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:35:01,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:35:01,078 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:35:01,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:35:01,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:01,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:35:01,121 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:35:01,121 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:35:01,125 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:35:01,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:35:01,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:35:01,641 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:35:01,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:35:01,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:35:01,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:35:01,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:35:01,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:35:01,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:35:01,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:35:01,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:35:01,782 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:35:01,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:35:01,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:01,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:01,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:35:01,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:35:01,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:01,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:01,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:35:01,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:35:02,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:35:02,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:35:02,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:35:02,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:35:02,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:35:02,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:35:02,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:35:02,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:35:02,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:35:02,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:35:02,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:02,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:02,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:02,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:02,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:02,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:02,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:02,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:02,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:02,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:02,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:02,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:02,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:02,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:02,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:02,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:02,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:02,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:02,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:03,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:35:03,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:35:03,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:03,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:35:03,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:35:03,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:03,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:35:03,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:35:03,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:03,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:35:03,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:35:03,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:03,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:35:03,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:35:03,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:35:03,357 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:35:03,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:35:03,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:03,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:35:03,391 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:35:03,391 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:35:03,394 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:35:03,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:35:03,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:35:03,924 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:35:03,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:35:03,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:35:03,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:35:03,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:35:04,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:35:04,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:35:04,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:35:04,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:35:04,084 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:35:04,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:35:04,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:35:04,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:35:04,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:35:04,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:35:04,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:35:04,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:35:04,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:35:04,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:35:04,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:35:04,818 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:35:04,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:04,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:04,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:35:04,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:35:04,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:35:05,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:35:05,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:35:05,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:35:05,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:35:05,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:35:05,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:35:05,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:35:05,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:05,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:35:05,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:05,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:35:05,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:05,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:35:05,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:05,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:35:05,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:05,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:35:05,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:05,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:35:05,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:05,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:05,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:35:05,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:05,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:35:06,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:06,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:35:06,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:35:06,061 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:35:06,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:35:06,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:35:06,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:35:06,104 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:35:06,104 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:35:06,104 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:35:06,107 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:35:06,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:35:06,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:35:06,634 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:35:06,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:35:06,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:35:06,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:35:06,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:35:06,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:35:06,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:35:06,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:35:06,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:06,895 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:35:06,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:06,901 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:08,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:09,073 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:35:09,078 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:09,078 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:11,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:11,249 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:35:11,253 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:11,253 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:13,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:13,424 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:35:13,430 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:13,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:15,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:15,596 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:35:15,602 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:15,602 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:17,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:17,750 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:17,750 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:19,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:19,904 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:19,904 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:21,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:22,053 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:22,053 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:24,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:24,204 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:24,204 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:26,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:26,346 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:26,346 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:28,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:28,492 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:28,493 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:30,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:30,640 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:30,640 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:32,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:32,852 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:32,854 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:34,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:35,015 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:35,015 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:37,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:37,174 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:37,174 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:39,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:39,345 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:39,345 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:41,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:41,502 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:41,502 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:43,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:43,651 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:43,651 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:45,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:45,803 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:45,803 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:47,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:47,960 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:47,960 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:49,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:50,114 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:50,115 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:52,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:52,278 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:52,278 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:54,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:54,430 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:54,431 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:56,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:56,594 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:56,595 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:35:58,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:35:58,755 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:35:58,756 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:00,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:00,910 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:00,911 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:02,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:03,071 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:03,071 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:05,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:05,229 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:05,229 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:07,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:07,387 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:07,387 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:09,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:09,562 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:09,563 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:11,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:12,066 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:12,066 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:14,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:14,430 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:14,430 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:16,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:16,690 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:16,691 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:18,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:18,938 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:18,938 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:20,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:21,535 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:21,535 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:23,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:23,698 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:23,698 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:25,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:25,974 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:25,974 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:36:27,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:28,273 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:36:28,273 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 82.1689999104. 2021-12-16 10:36:28,275 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:36:28,276 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:36:28,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:36:28,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:36:28,790 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:36:28,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:36:28,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:36:28,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:36:28,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:36:28,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:36:28,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:36:28,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:36:28,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:29,072 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:36:29,072 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.797999858856. 2021-12-16 10:36:29,072 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:36:29,075 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:36:29,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:36:29,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:36:29,607 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:36:29,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:36:29,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:36:29,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:36:29,665 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:36:29,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:36:29,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:36:29,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:36:29,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:29,903 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:36:29,903 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.830000162125. 2021-12-16 10:36:29,904 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:36:29,905 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:36:29,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:36:30,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:36:30,230 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:36:30,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:36:30,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:36:30,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:36:30,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:36:30,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:36:30,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:36:30,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:36:30,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:30,490 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:30,492 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:36:32,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:32,651 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:32,651 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:36:34,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:34,813 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:36:34,813 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:36:36,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:36:37,265 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:36:37,265 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 7.36000013351. ############################################################ # # RAM FootPrint: 4127.6953125 # ############################################################ 2021-12-16 10:36:37,298 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:36:37,298 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 111 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-16 10:38:28,302 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:38:28,302 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:38:33,305 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:38:33,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:33,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:38:33,733 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:33,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:38:33,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:38:33,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:38:33,772 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:38:33,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:38:33,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:38:33,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:38:33,825 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:38:33,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:38:33,874 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['268283 memcached\n'], []) 2021-12-16 10:38:33,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:38:33,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:38:34,009 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['268283 memcached\n'], []) 2021-12-16 10:38:34,009 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:38:34,013 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:38:34,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:34,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:38:34,401 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:34,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:38:34,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:38:34,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:38:34,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:38:34,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:38:34,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:38:34,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:38:34,490 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:38:34,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:38:34,542 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['83512 memcached\n'], []) 2021-12-16 10:38:34,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:38:34,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:38:34,625 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['83512 memcached\n'], []) 2021-12-16 10:38:34,625 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:38:34,628 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:38:34,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:34,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:38:35,061 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:35,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:38:35,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:38:35,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:38:35,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:38:35,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:38:35,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:38:35,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:38:35,181 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:38:35,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:38:35,263 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['171173 memcached\n'], []) 2021-12-16 10:38:35,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:38:35,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:38:35,391 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['171173 memcached\n'], []) 2021-12-16 10:38:35,391 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:38:35,394 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:38:35,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:35,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:38:35,997 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:35,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:38:36,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:38:36,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:38:36,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:38:36,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:38:36,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:38:36,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:38:36,157 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:38:36,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:38:36,227 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['6368 memcached\n'], []) 2021-12-16 10:38:36,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:38:36,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:38:36,375 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['6368 memcached\n'], []) 2021-12-16 10:38:36,375 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:38:36,378 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:38:36,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:40,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:38:40,155 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:40,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:38:40,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:38:40,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:38:40,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:38:40,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:38:40,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:38:40,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:38:40,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:38:40,302 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:38:40,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:38:40,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:40,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:40,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:38:40,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:38:40,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:40,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:40,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:38:40,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:38:40,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:40,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:40,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:38:40,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:38:40,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:40,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:40,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:40,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:40,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:40,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:40,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:41,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:41,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:41,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:41,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:41,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:41,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:41,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:41,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:41,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:41,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:38:41,301 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:38:41,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:38:41,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:41,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:38:41,345 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:38:41,345 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:38:41,348 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:38:41,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:41,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:38:41,790 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:41,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:38:41,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:38:41,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:38:41,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:38:41,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:38:41,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:38:41,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:38:41,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:38:41,913 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:38:41,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:38:41,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,038 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:42,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:42,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:42,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:42,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:42,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:42,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:42,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:42,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:42,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:42,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:42,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:42,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:42,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:42,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:42,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:42,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:43,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:43,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:43,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:43,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:43,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:43,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:43,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:43,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:43,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:43,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:43,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:43,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:43,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:43,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:38:43,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:38:43,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:38:43,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:38:43,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:43,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:43,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:38:44,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:38:44,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:44,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:44,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:38:44,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:38:44,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:44,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:44,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:38:44,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:38:44,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:44,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:44,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:38:44,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:38:44,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:38:44,256 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:38:44,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:38:44,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:44,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:38:44,303 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:38:44,303 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:38:44,306 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:38:44,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:44,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:38:44,826 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:44,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:38:44,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:38:44,862 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:38:44,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:38:44,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:38:44,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:38:44,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:38:44,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:38:44,984 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:38:45,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:38:45,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:45,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:45,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:45,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:45,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,395 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:45,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:45,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:45,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:45,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:45,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:45,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:45,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:45,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:45,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:45,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:46,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:46,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:46,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:46,198 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:46,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:46,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:46,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:46,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:46,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:46,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:46,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:46,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:46,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:46,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:46,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:46,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:47,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:47,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:47,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:47,214 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:38:47,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:38:47,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:38:47,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:38:47,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:38:47,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:38:47,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:38:47,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:38:47,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:47,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:38:47,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:38:47,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:38:47,957 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:38:47,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:38:47,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:47,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:38:48,006 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:38:48,007 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:38:48,010 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:38:48,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:48,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:38:48,516 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:48,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:38:48,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:38:48,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:38:48,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:38:48,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:38:48,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:38:48,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:38:48,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:38:48,673 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:38:48,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:38:48,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:48,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:48,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:38:48,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:38:48,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:48,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:48,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:38:48,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:38:49,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,038 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:38:49,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:38:49,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:49,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:38:49,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:49,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:38:49,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:49,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:38:49,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:49,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:38:49,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:49,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:38:49,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:49,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:38:49,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:49,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:38:49,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:49,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:38:49,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:49,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:49,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:50,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:38:50,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:50,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:50,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:50,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:38:50,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:50,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:50,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:50,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:38:50,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:51,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:51,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:51,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:38:51,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:51,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:51,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:51,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:38:51,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:51,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:51,318 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:51,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:38:51,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:51,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:38:51,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:51,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:38:51,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:38:51,486 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:38:51,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:38:51,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:38:51,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:38:51,535 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:38:51,535 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:38:51,535 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:38:51,538 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:38:51,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:38:51,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:38:52,055 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:38:52,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:38:52,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:38:52,094 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:38:52,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:38:52,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:38:52,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:38:52,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:38:52,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:38:52,334 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:38:52,336 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:38:52,338 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:38:54,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:38:54,519 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:38:54,523 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:38:54,523 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:38:56,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:38:56,694 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:38:56,700 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:38:56,700 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:38:58,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:38:58,868 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:38:58,874 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:38:58,874 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:00,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:01,049 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:39:01,055 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:01,055 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:03,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:03,234 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:39:03,240 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:03,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:05,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:05,403 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:39:05,407 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:05,407 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:07,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:07,592 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:39:07,595 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:07,595 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:09,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:09,779 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:39:09,782 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:09,783 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:11,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:11,924 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:11,924 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:13,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:14,066 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:14,066 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:16,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:16,216 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:16,216 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:18,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:18,358 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:18,358 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:20,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:20,503 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:20,503 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:22,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:22,665 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:22,665 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:24,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:24,821 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:24,821 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:26,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:26,960 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:26,960 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:28,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:29,111 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:29,111 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:31,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:31,263 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:31,263 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:33,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:33,408 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:33,408 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:35,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:35,568 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:35,568 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:37,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:37,729 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:37,729 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:39,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:39,880 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:39,880 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:41,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:42,039 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:42,039 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:44,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:44,186 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:44,186 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:46,186 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:46,355 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:46,355 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:48,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:48,503 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:48,503 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:50,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:50,657 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:50,657 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:52,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:52,809 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:52,809 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:54,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:54,970 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:54,970 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:56,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:57,130 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:57,130 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:39:59,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:39:59,292 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:39:59,292 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:01,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:01,447 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:01,447 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:03,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:03,598 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:03,599 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:05,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:05,756 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:05,756 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:07,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:07,914 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:07,914 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:09,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:10,073 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:10,073 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:12,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:12,234 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:12,236 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:14,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:14,451 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:14,451 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:16,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:16,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:16,901 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:18,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:19,059 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:19,059 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:21,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:21,289 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:21,289 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:23,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:23,477 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:40:23,477 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:40:25,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:26,434 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:40:26,436 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 94.9010000229. 2021-12-16 10:40:26,437 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:40:26,437 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:40:26,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:40:26,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:40:26,956 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:40:26,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:40:26,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:40:26,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:40:27,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:40:27,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:40:27,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:40:27,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:40:27,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:27,293 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:40:27,293 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.856999874115. 2021-12-16 10:40:27,296 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:40:27,296 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:40:27,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:40:27,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:40:27,808 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:40:27,808 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:40:27,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:40:27,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:40:27,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:40:27,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:40:27,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:40:27,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:40:27,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:28,066 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:40:28,068 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.772000074387. 2021-12-16 10:40:28,068 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:40:28,069 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:40:28,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:40:28,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:40:28,512 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:40:28,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:40:28,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:40:28,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:40:28,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:40:28,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:40:28,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:40:28,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:40:28,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:40:28,759 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:40:28,759 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.691999912262. ############################################################ # # RAM FootPrint: 4056.09765625 # ############################################################ 2021-12-16 10:40:28,792 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:40:28,792 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 92 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-16 10:42:00,796 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:42:00,798 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:42:05,799 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:42:05,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:06,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:42:06,345 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:06,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:42:06,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:42:06,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:42:06,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:42:06,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:42:06,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:42:06,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:42:06,444 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:42:06,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:42:06,500 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['269390 memcached\n'], []) 2021-12-16 10:42:06,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:42:06,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:42:06,588 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['269390 memcached\n'], []) 2021-12-16 10:42:06,589 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:42:06,592 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:42:06,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:06,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:42:06,980 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:06,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:42:06,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:42:07,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:42:07,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:42:07,043 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:42:07,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:42:07,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:42:07,084 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:42:07,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:42:07,134 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['84484 memcached\n'], []) 2021-12-16 10:42:07,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:42:07,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:42:07,242 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['84484 memcached\n'], []) 2021-12-16 10:42:07,242 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:42:07,246 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:42:07,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:07,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:42:07,763 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:07,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:42:07,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:42:07,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:42:07,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:42:07,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:42:07,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:42:07,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:42:07,888 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:42:07,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:42:07,938 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['172459 memcached\n'], []) 2021-12-16 10:42:07,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:42:07,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:42:08,033 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['172459 memcached\n'], []) 2021-12-16 10:42:08,033 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:42:08,038 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:42:08,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:08,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:42:08,507 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:08,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:42:08,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:42:08,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:42:08,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:42:08,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:42:08,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:42:08,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:42:08,782 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:42:08,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:42:08,859 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['8006 memcached\n'], []) 2021-12-16 10:42:08,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:42:08,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:42:09,006 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['8006 memcached\n'], []) 2021-12-16 10:42:09,006 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:42:09,009 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:42:09,130 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:09,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:42:09,487 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:09,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:42:09,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:42:09,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:42:09,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:42:09,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:42:09,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:42:09,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:42:09,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:42:09,648 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:42:09,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:42:09,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:09,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:09,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:42:09,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:42:09,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:09,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:09,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:42:09,941 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:42:09,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:09,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:42:10,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:42:10,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:10,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:10,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:10,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:10,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:10,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:10,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:10,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:10,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:10,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:10,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:10,709 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:42:10,723 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:42:10,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:42:10,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:10,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:42:10,766 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:42:10,766 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:42:10,769 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:42:10,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:11,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:42:11,250 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:11,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:42:11,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:42:11,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:42:11,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:42:11,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:42:11,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:42:11,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:42:11,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:42:11,403 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:42:11,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:42:11,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:11,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:11,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:11,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:11,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:11,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:11,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:11,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:11,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:11,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:11,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:11,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:11,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:11,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:11,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:11,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:11,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:12,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:12,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:12,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:12,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:12,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:12,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:12,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:12,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:12,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:12,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:12,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:12,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:12,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:12,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:12,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:12,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:13,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:13,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:13,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:13,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:13,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:13,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:42:13,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:42:13,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:42:13,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:42:13,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:42:13,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:42:13,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:42:13,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:42:13,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:42:13,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:42:13,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:13,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:42:13,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:42:13,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:42:13,917 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:42:13,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:42:13,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:13,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:42:13,956 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:42:13,956 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:42:13,959 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:42:14,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:14,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:42:14,460 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:14,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:42:14,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:42:14,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:42:14,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:42:14,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:42:14,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:42:14,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:42:14,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:42:14,611 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:42:14,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:42:14,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:14,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:14,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:14,724 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:14,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:14,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:14,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:14,818 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:14,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:14,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:14,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:14,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:14,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:14,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:14,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:15,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:15,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:15,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:15,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:15,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:15,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:15,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:15,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:15,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:15,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:15,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:15,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:15,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:15,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:15,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:15,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:15,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:15,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:15,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:15,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:16,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:16,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:16,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:16,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:16,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:16,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:42:16,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:42:16,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:42:16,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:42:16,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:42:16,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:42:16,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:42:16,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:42:16,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:16,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:42:16,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:42:16,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:42:16,897 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:42:16,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:42:16,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:16,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:42:16,938 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:42:16,938 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:42:16,940 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:42:17,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:17,354 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:42:17,433 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:17,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:42:17,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:42:17,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:42:17,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:42:17,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:42:17,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:42:17,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:42:17,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:42:17,575 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:42:17,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:42:17,668 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:17,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:17,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:42:17,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:42:17,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:17,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:17,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:42:17,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:42:17,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:17,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:42:18,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:42:18,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:18,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:42:18,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:18,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:42:18,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:18,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:42:18,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:18,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:42:18,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:18,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:42:18,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:18,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:42:18,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:18,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:42:18,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:18,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:42:18,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:18,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:18,984 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:19,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:42:19,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:19,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:42:19,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:19,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:42:19,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:19,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:42:19,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:19,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:42:19,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:19,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:42:19,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:42:19,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:19,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:42:19,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:42:19,812 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:42:19,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:42:19,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:42:19,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:42:19,859 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:42:19,861 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:42:19,861 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:42:19,864 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:42:19,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:42:20,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:42:20,407 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:42:20,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:42:20,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:42:20,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:42:20,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:42:20,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:42:20,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:42:20,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:42:20,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:20,732 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:20,736 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:20,736 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:22,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:22,910 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:22,914 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:22,914 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:24,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:25,088 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:25,095 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:25,095 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:27,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:27,265 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:27,301 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:27,301 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:29,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:29,470 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:29,474 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:29,474 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:31,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:31,651 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:31,657 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:31,657 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:33,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:33,835 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:42:33,838 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:33,838 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:35,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:35,996 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:35,996 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:37,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:38,145 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:38,145 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:40,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:40,303 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:40,303 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:42,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:42,482 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:42,482 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:44,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:44,634 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:44,634 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:46,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:46,780 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:46,782 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:48,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:48,934 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:48,934 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:50,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:51,082 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:51,084 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:53,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:53,233 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:53,234 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:55,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:55,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:55,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:57,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:57,532 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:57,532 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:42:59,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:42:59,700 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:42:59,700 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:01,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:01,854 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:01,855 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:03,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:04,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:04,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:06,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:06,171 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:06,171 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:08,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:08,329 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:08,329 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:10,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:10,490 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:10,490 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:12,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:12,650 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:12,651 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:14,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:14,816 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:14,816 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:16,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:16,976 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:16,976 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:18,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:19,130 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:19,130 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:21,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:21,286 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:21,286 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:23,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:23,440 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:23,440 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:25,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:25,592 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:25,592 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:27,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:27,747 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:27,749 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:29,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:29,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:29,901 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:31,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:32,066 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:32,066 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:34,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:34,226 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:34,226 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:36,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:36,388 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:36,388 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:38,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:38,539 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:38,539 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:40,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:40,729 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:40,729 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:42,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:42,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:42,901 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:44,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:45,071 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:45,071 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:47,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:47,233 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:47,233 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:43:49,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:49,411 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:43:49,411 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 89.5499999523. 2021-12-16 10:43:49,411 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:43:49,414 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:43:49,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:43:49,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:43:49,936 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:43:49,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:43:49,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:43:49,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:43:49,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:43:50,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:43:50,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:43:50,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:43:50,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:50,223 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:43:50,223 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.81200003624. 2021-12-16 10:43:50,223 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:43:50,226 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:43:50,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:43:50,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:43:50,753 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:43:50,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:43:50,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:43:50,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:43:50,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:43:50,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:43:50,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:43:50,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:43:50,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:51,038 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:43:51,038 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.8140001297. 2021-12-16 10:43:51,038 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:43:51,040 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:43:51,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:43:51,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:43:51,392 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:43:51,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:43:51,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:43:51,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:43:51,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:43:51,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:43:51,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:43:51,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:43:51,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:51,674 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:51,674 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:43:53,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:53,838 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:43:53,838 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 10:43:55,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:43:56,263 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:43:56,263 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.22599983215. ############################################################ # # RAM FootPrint: 3966.12109375 # ############################################################ 2021-12-16 10:43:56,295 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:43:56,295 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-16 10:45:39,299 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:45:39,299 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:45:44,301 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:45:44,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:44,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:45:44,959 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:44,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:45:44,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:45:44,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:45:45,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:45:45,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:45:45,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:45:45,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:45:45,088 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:45:45,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:45:45,180 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['271729 memcached\n'], []) 2021-12-16 10:45:45,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:45:45,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:45:45,311 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['271729 memcached\n'], []) 2021-12-16 10:45:45,311 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:45:45,313 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:45:45,436 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:45,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:45:45,802 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:45,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:45:45,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:45:45,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:45:45,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:45:45,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:45:45,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:45:45,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:45:45,950 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:45:45,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:45:46,032 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['85475 memcached\n'], []) 2021-12-16 10:45:46,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:45:46,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:45:46,168 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['85475 memcached\n'], []) 2021-12-16 10:45:46,170 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:45:46,171 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:45:46,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:46,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:45:46,700 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:46,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:45:46,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:45:46,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:45:46,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:45:46,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:45:46,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:45:46,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:45:46,834 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:45:46,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:45:46,894 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['173733 memcached\n'], []) 2021-12-16 10:45:46,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:45:46,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:45:47,012 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['173733 memcached\n'], []) 2021-12-16 10:45:47,012 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:45:47,013 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:45:47,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:47,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:45:47,480 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:47,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:45:47,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:45:47,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:45:47,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:45:47,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:45:47,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:45:47,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:45:47,664 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:45:47,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:45:47,756 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['9593 memcached\n'], []) 2021-12-16 10:45:47,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:45:47,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:45:47,911 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['9593 memcached\n'], []) 2021-12-16 10:45:47,911 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:45:47,914 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:45:47,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:48,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:45:48,375 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:48,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:45:48,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:45:48,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:45:48,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:45:48,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:45:48,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:45:48,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:45:48,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:45:48,503 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:45:48,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:45:48,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:48,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:48,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:45:48,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:45:48,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:48,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:48,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:45:48,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:45:48,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:48,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:48,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:45:48,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:45:48,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:48,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:48,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:48,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:48,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:48,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:49,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:49,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:49,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:49,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:49,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:49,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:49,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:49,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:49,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:49,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:49,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:49,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:49,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:49,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:49,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:45:49,351 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:45:49,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:45:49,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:49,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:45:49,397 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:45:49,397 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:45:49,400 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:45:49,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:49,747 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:45:49,835 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:49,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:45:49,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:45:49,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:45:49,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:45:49,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:45:49,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:45:49,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:45:49,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:45:50,000 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:45:50,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:45:50,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:50,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:50,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:50,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:50,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:50,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:50,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:50,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:50,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:50,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:50,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:50,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:50,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:50,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:50,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:50,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:50,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,772 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:50,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:50,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:50,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:50,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:50,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:50,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:50,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:50,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:51,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:51,020 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:51,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:51,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:51,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:51,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:51,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:51,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:45:51,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:45:51,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:45:51,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:45:51,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:45:51,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:45:51,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:45:51,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:45:51,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:51,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:45:51,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:45:51,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:45:51,601 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:45:51,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:45:51,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:51,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:45:51,638 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:45:51,638 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:45:51,640 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:45:51,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:52,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:45:52,118 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:52,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:45:52,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:45:52,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:45:52,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:45:52,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:45:52,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:45:52,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:45:52,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:45:52,229 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:45:52,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:45:52,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:52,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:52,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,436 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:52,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:52,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:52,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:52,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:52,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:52,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:52,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:52,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:52,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:52,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:53,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:53,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:53,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:53,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:53,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:53,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:53,334 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:53,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:53,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:53,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:53,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:53,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:53,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:53,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:53,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:53,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:53,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:53,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:53,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:54,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:54,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:54,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,214 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:54,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:54,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:45:54,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:45:54,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:45:54,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:45:54,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:45:54,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:45:54,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:54,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:45:54,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:45:54,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:45:54,673 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:45:54,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:45:54,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:54,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:45:54,707 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:45:54,707 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:45:54,710 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:45:54,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:55,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:45:55,229 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:55,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:45:55,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:45:55,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:45:55,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:45:55,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:45:55,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:45:55,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:45:55,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:45:55,371 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:45:55,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:45:55,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:55,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:55,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:45:55,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:45:55,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:55,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:55,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:45:55,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:45:55,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:55,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:55,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:45:55,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:45:55,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:55,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:55,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:55,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:45:55,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:56,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:45:56,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,186 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:56,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:45:56,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:56,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:45:56,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:56,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:45:56,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:56,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:45:56,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:56,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:45:56,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:56,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:45:56,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:56,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:45:56,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,795 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:56,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:45:56,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:56,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:56,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:56,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:45:57,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:57,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:57,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:45:57,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:57,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:57,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:45:57,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:57,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:57,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:45:57,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:57,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:57,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:45:57,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:45:57,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:57,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:45:57,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:45:57,571 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:45:57,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:45:57,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:45:57,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:45:57,622 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:45:57,622 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:45:57,622 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:45:57,625 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:45:57,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:45:57,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:45:58,058 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:45:58,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:45:58,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:45:58,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:45:58,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:45:58,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:45:58,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:45:58,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:45:58,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:45:58,410 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:45:58,414 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:45:58,414 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:00,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:00,578 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:46:00,582 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:00,582 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:02,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:02,727 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:46:02,733 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:02,733 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:04,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:04,914 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:46:04,917 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:04,917 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:06,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:07,098 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:46:07,101 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:07,101 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:09,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:09,270 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:46:09,275 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:09,275 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:11,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:11,470 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:11,470 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:13,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:13,619 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:13,619 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:15,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:15,769 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:15,769 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:17,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:17,930 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:17,930 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:19,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:20,095 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:20,095 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:22,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:22,252 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:22,252 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:24,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:24,394 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:24,394 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:26,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:26,538 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:26,539 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:28,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:28,691 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:28,691 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:30,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:30,852 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:30,852 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:32,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:33,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:33,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:35,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:35,177 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:35,177 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:37,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:37,335 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:37,335 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:39,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:39,483 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:39,483 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:41,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:41,641 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:41,641 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:43,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:43,801 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:43,801 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:45,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:45,950 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:45,951 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:47,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:48,112 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:48,112 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:50,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:50,262 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:50,262 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:52,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:52,420 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:52,420 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:54,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:54,566 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:54,566 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:56,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:56,726 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:56,726 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:46:58,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:46:58,888 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:46:58,888 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:00,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:01,039 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:01,039 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:03,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:03,197 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:03,197 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:05,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:05,342 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:05,344 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:07,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:07,494 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:07,494 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:09,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:09,673 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:09,673 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:11,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:11,845 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:11,845 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:13,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:14,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:14,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:16,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:16,187 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:16,187 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:18,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:18,361 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:18,361 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:20,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:20,523 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:20,523 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:22,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:22,693 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:47:22,693 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:47:24,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:24,947 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:47:24,947 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.3250000477. 2021-12-16 10:47:24,950 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:47:24,951 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:47:25,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:47:25,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:47:25,450 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:47:25,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:47:25,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:47:25,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:47:25,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:47:25,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:47:25,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:47:25,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:47:25,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:25,848 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:47:25,849 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.899999856949. 2021-12-16 10:47:25,851 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:47:25,852 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:47:25,966 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:47:26,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:47:26,365 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:47:26,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:47:26,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:47:26,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:47:26,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:47:26,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:47:26,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:47:26,451 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:47:26,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:26,602 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:47:26,604 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.753000020981. 2021-12-16 10:47:26,604 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:47:26,605 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:47:26,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:47:27,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:47:27,378 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:47:27,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:47:27,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:47:27,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:47:27,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:47:27,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:47:27,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:47:27,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:47:27,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:47:28,279 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:47:28,279 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.67599987984. ############################################################ # # RAM FootPrint: 3911.58984375 # ############################################################ 2021-12-16 10:47:28,316 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:47:28,316 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 89 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-16 10:48:57,321 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:48:57,321 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:49:02,322 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:49:02,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:02,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:49:02,851 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:02,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:49:02,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:49:02,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:49:02,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:49:02,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:49:02,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:49:02,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:49:02,986 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:49:02,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:49:03,055 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['273731 memcached\n'], []) 2021-12-16 10:49:03,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:49:03,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:49:03,186 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['273731 memcached\n'], []) 2021-12-16 10:49:03,186 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:49:03,188 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:49:03,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:03,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:49:03,684 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:03,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:49:03,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:49:03,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:49:03,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:49:03,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:49:03,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:49:03,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:49:03,842 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:49:03,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:49:03,921 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['86452 memcached\n'], []) 2021-12-16 10:49:03,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:49:03,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:49:04,035 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['86452 memcached\n'], []) 2021-12-16 10:49:04,036 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:49:04,039 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:49:04,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:04,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:49:04,566 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:04,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:49:04,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:49:04,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:49:04,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:49:04,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:49:04,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:49:04,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:49:04,691 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:49:04,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:49:04,762 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['175009 memcached\n'], []) 2021-12-16 10:49:04,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:49:04,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:49:04,884 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['175009 memcached\n'], []) 2021-12-16 10:49:04,885 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:49:04,888 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:49:04,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:05,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:49:05,342 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:05,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:49:05,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:49:05,385 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:49:05,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:49:05,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:49:05,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:49:05,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:49:05,500 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:49:05,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:49:05,586 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['11198 memcached\n'], []) 2021-12-16 10:49:05,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:49:05,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:49:05,740 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['11198 memcached\n'], []) 2021-12-16 10:49:05,740 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:49:05,743 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:49:05,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:12,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:49:12,128 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:12,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:49:12,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:49:12,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:49:12,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:49:12,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:49:12,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:49:12,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:49:12,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:49:12,302 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:49:12,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:49:12,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:49:12,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:49:12,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:49:12,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:49:12,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:49:12,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:49:12,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:12,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:12,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:12,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:12,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:12,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:12,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:12,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:12,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:12,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:12,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:13,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:13,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:49:13,042 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:49:13,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:49:13,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:13,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:49:13,088 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:49:13,088 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:49:13,092 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:49:13,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:13,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:49:13,558 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:13,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:49:13,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:49:13,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:49:13,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:49:13,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:49:13,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:49:13,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:49:13,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:49:13,714 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:49:13,744 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:49:13,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:13,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:13,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:13,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:13,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:13,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:13,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:13,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:13,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:13,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:14,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:14,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:14,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:14,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:14,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:14,186 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:14,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:14,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:14,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:14,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:14,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:14,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:14,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:14,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:14,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:14,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:14,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:14,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:14,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:14,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:14,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:14,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:14,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:14,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:14,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:14,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:15,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:15,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:15,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:49:15,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:49:15,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:15,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:49:15,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:49:15,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:15,400 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:49:15,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:49:15,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:15,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:49:15,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:49:15,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:15,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:49:15,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:49:15,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:49:15,688 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:49:15,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:49:15,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:15,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:49:15,750 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:49:15,750 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:49:15,753 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:49:15,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:16,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:49:16,278 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:16,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:49:16,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:49:16,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:49:16,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:49:16,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:49:16,395 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:49:16,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:49:16,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:49:16,463 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:49:16,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:49:16,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:16,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:16,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:16,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:16,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:16,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:16,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:16,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:16,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:16,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:16,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:16,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:16,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:16,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:16,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:17,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:17,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:17,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:17,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:17,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:17,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:17,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:17,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:17,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:17,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:17,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:17,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,772 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:17,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:17,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:17,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:17,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:17,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:17,977 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:18,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:18,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:18,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:18,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:18,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:18,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:18,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:18,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:18,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:18,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:49:18,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:49:18,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:49:18,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:49:18,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:18,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:18,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:49:19,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:49:19,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:19,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:19,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:49:19,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:49:19,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:49:19,161 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:49:19,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:49:19,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:19,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:49:19,216 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:49:19,216 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:49:19,219 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:49:19,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:19,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:49:20,108 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:20,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:49:20,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:49:20,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:49:20,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:49:20,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:49:20,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:49:20,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:49:20,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:49:20,697 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:49:20,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:49:20,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:20,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:21,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:49:21,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:49:21,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:21,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:21,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:49:21,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:49:21,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:21,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:21,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:49:21,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:49:21,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:21,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:21,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:22,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:49:22,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:22,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:22,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:22,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:49:22,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:22,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:22,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:22,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:49:22,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:22,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:23,048 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:23,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:49:23,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:23,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:23,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:23,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:49:23,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:23,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:23,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:23,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:49:23,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:23,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:23,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:24,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:49:24,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:24,130 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:24,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:24,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:49:24,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:24,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:24,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:24,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:49:24,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:24,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:24,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:24,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:49:25,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:25,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:25,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:25,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:49:25,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:25,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:25,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:25,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:49:25,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:25,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:25,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:25,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:49:26,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:26,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:26,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:26,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:49:26,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:26,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:26,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:26,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:49:26,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:26,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:49:27,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:27,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:49:27,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:49:27,267 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:49:27,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:49:27,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:49:27,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:49:27,346 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:49:27,346 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:49:27,346 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:49:27,349 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:49:27,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:49:27,775 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:49:27,858 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:49:27,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:49:27,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:49:27,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:49:27,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:49:27,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:49:27,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:49:27,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:49:27,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:28,144 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:49:28,148 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:28,148 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:30,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:30,295 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:30,296 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:32,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:32,444 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:32,444 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:34,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:34,588 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:34,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:36,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:36,727 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:36,729 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:38,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:38,869 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:38,869 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:40,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:41,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:41,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:43,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:43,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:43,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:45,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:45,325 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:45,325 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:47,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:47,482 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:47,482 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:49,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:49,631 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:49,632 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:51,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:51,789 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:51,789 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:53,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:53,956 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:53,957 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:55,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:56,101 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:56,101 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:49:58,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:49:58,257 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:49:58,257 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:00,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:00,407 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:00,407 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:02,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:02,569 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:02,569 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:04,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:04,734 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:04,734 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:06,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:06,890 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:06,890 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:08,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:09,043 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:09,043 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:11,043 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:11,204 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:11,204 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:13,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:13,355 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:13,355 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:15,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:15,506 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:15,506 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:17,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:17,655 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:17,657 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:19,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:19,812 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:19,812 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:21,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:21,973 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:21,973 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:23,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:24,132 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:24,132 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:26,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:26,289 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:26,289 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:28,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:28,987 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:28,987 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:30,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:31,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:31,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:33,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:33,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:33,348 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:35,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:35,519 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:35,519 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:37,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:37,678 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:37,678 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:39,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:39,841 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:50:39,842 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:50:41,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:42,061 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:50:42,062 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 74.7150001526. 2021-12-16 10:50:42,062 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:50:42,065 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:50:42,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:50:42,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:50:42,586 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:50:42,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:50:42,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:50:42,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:50:42,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:50:42,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:50:42,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:50:42,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:50:42,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:42,960 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:50:42,961 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.898999929428. 2021-12-16 10:50:42,961 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:50:42,963 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:50:43,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:50:43,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:50:43,460 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:50:43,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:50:43,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:50:43,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:50:43,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:50:43,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:50:43,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:50:43,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:50:43,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:43,676 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:50:43,677 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.715000152588. 2021-12-16 10:50:43,677 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:50:43,678 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:50:43,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:50:44,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:50:44,203 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:50:44,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:50:44,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:50:44,298 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:50:44,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:50:44,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:50:44,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:50:44,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:50:44,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:50:45,061 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:50:45,061 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.38300013542. ############################################################ # # RAM FootPrint: 3856.484375 # ############################################################ 2021-12-16 10:50:45,094 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:50:45,094 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 88 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-16 10:52:13,098 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:52:13,098 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:52:18,101 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:52:18,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:18,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:52:18,947 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:18,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:52:18,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:52:18,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:52:19,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:52:19,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:52:19,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:52:19,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:52:19,177 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:52:19,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:52:19,253 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['274837 memcached\n'], []) 2021-12-16 10:52:19,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:52:19,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:52:19,430 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['274837 memcached\n'], []) 2021-12-16 10:52:19,431 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:52:19,434 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:52:19,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:19,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:52:19,920 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:19,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:52:19,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:52:19,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:52:19,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:52:20,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:52:20,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:52:20,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:52:20,072 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:52:20,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:52:20,155 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['87428 memcached\n'], []) 2021-12-16 10:52:20,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:52:20,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:52:20,302 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['87428 memcached\n'], []) 2021-12-16 10:52:20,302 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:52:20,305 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:52:20,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:20,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:52:20,826 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:20,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:52:20,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:52:20,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:52:20,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:52:20,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:52:20,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:52:20,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:52:20,957 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:52:20,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:52:21,009 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['176292 memcached\n'], []) 2021-12-16 10:52:21,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:52:21,053 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:52:21,115 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['176292 memcached\n'], []) 2021-12-16 10:52:21,115 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:52:21,118 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:52:21,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:21,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:52:21,582 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:21,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:52:21,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:52:21,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:52:21,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:52:21,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:52:21,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:52:21,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:52:21,757 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:52:21,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:52:21,842 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['12566 memcached\n'], []) 2021-12-16 10:52:21,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:52:21,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:52:21,983 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['12566 memcached\n'], []) 2021-12-16 10:52:21,983 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:52:21,986 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:52:22,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:24,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:52:24,450 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:24,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:52:24,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:52:24,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:52:24,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:52:24,522 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:52:24,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:52:24,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:52:24,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:52:24,582 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:52:24,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:52:24,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:24,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:24,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:52:24,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:52:24,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:24,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:24,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:52:24,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:52:24,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:24,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:24,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:52:25,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:52:25,030 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:25,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:25,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:25,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:25,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:25,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:25,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:25,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:25,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:25,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:25,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:25,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:25,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,403 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:25,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:25,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:25,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:52:25,467 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:52:25,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:52:25,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:25,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:52:25,516 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:52:25,516 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:52:25,519 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:52:25,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:25,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:52:25,976 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:25,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:52:25,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:52:26,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:52:26,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:52:26,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:52:26,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:52:26,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:52:26,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:52:26,121 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:52:26,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:52:26,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:26,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:26,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:26,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,365 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:26,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:26,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:26,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:26,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:26,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:26,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,665 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:26,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:26,742 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:26,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:26,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:26,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:26,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:26,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:27,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:27,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:27,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:27,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:27,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:27,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:27,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:27,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:27,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:27,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:27,792 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:27,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:27,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:27,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:27,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:27,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:28,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:28,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:28,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:28,234 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:28,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:28,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,418 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:52:28,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:52:28,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:52:28,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:52:28,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:52:28,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:52:28,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:52:28,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:52:28,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:28,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:52:28,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:52:28,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:52:28,851 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:52:28,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:52:28,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:28,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:52:28,894 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:52:28,895 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:52:28,898 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:52:29,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:29,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:52:29,407 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:29,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:52:29,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:52:29,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:52:29,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:52:29,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:52:29,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:52:29,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:52:29,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:52:29,549 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:52:29,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:52:29,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:29,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:29,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:29,696 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:29,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:29,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:29,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:29,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:29,823 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:29,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:29,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:29,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:29,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:29,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:30,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:30,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:30,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:30,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:30,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:30,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:30,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:30,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:30,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:30,663 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:30,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:30,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:30,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:30,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:30,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:30,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:31,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:31,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:31,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:31,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,306 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:31,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:31,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:31,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:31,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:31,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:31,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:31,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:31,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,704 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:52:31,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:52:31,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:52:31,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:52:31,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:31,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:31,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:52:32,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:52:32,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:32,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:32,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:52:32,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:52:32,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:52:32,168 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:52:32,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:52:32,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:32,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:52:32,217 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:52:32,217 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:52:32,220 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:52:32,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:32,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:52:32,746 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:32,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:52:32,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:52:32,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:52:32,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:52:32,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:52:32,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:52:32,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:52:32,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:52:32,937 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:52:32,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:52:32,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:52:33,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:52:33,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:52:33,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:52:33,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:52:33,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:52:33,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:33,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:52:33,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:33,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:52:33,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:33,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:52:33,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:33,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:52:33,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:33,943 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:52:33,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:33,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:33,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:34,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:52:34,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:34,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:52:34,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,121 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:34,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:52:34,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:34,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:52:34,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:34,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:52:34,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:34,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:52:34,415 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:34,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:52:34,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:34,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:52:34,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:34,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:52:34,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:34,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:52:34,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:52:34,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:34,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:52:34,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:52:34,826 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:52:34,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:52:34,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:52:34,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:52:34,871 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:52:34,871 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:52:34,871 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:52:34,875 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:52:34,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:52:35,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:52:35,387 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:52:35,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:52:35,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:52:35,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:52:35,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:52:35,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:52:35,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:52:35,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:52:35,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:35,638 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:35,642 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:35,644 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:37,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:37,823 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:37,829 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:37,829 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:39,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:39,996 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:40,000 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:40,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:42,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:42,165 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:42,170 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:42,171 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:44,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:44,348 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:44,354 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:44,354 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:46,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:46,529 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:52:46,532 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:46,533 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:48,533 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:48,681 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:48,683 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:50,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:50,835 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:50,835 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:52,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:52,982 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:52,983 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:54,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:55,134 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:55,134 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:57,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:57,283 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:57,283 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:52:59,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:52:59,434 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:52:59,434 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:01,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:01,575 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:01,575 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:03,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:03,737 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:03,739 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:05,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:05,908 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:05,908 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:07,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:08,055 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:08,055 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:10,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:10,214 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:10,214 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:12,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:12,371 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:12,371 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:14,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:14,532 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:14,532 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:16,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:16,690 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:16,690 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:18,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:18,849 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:18,849 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:20,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:21,010 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:21,010 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:23,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:23,171 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:23,173 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:25,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:25,331 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:25,331 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:27,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:27,486 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:27,486 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:29,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:29,647 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:29,648 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:31,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:31,808 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:31,808 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:33,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:33,961 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:33,961 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:35,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:36,114 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:36,114 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:38,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:38,276 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:38,278 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:40,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:40,444 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:40,446 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:42,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:42,592 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:42,592 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:44,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:45,086 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:45,086 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:47,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:47,267 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:47,267 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:49,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:49,428 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:49,428 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:51,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:51,716 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:51,716 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:53,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:53,874 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:53,874 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:55,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:56,033 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:56,035 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:53:58,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:53:58,210 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:53:58,210 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:54:00,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:54:00,388 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:54:00,388 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 85.5169999599. 2021-12-16 10:54:00,390 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:54:00,391 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:54:00,538 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:54:00,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:54:00,937 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:54:00,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:54:00,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:54:00,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:54:01,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:54:01,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:54:01,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:54:01,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:54:01,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:54:01,308 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:54:01,308 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.917999982834. 2021-12-16 10:54:01,309 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:54:01,311 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:54:01,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:54:01,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:54:01,848 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:54:01,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:54:01,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:54:01,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:54:01,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:54:01,928 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:54:01,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:54:01,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:54:01,977 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:54:02,141 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:54:02,141 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832000017166. 2021-12-16 10:54:02,141 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:54:02,144 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:54:02,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:54:02,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:54:02,753 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:54:02,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:54:02,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:54:02,882 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:54:02,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:54:02,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:54:03,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:54:03,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:54:03,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:54:03,464 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:54:03,466 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.32400012016. ############################################################ # # RAM FootPrint: 3819.625 # ############################################################ 2021-12-16 10:54:03,496 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:54:03,496 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-16 10:55:50,500 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:55:50,500 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:55:55,503 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:55:55,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:55:56,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:55:56,437 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:55:56,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:55:56,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:55:56,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:55:56,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:55:56,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:55:56,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:55:56,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:55:56,566 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:55:56,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:55:56,648 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['276563 memcached\n'], []) 2021-12-16 10:55:56,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:55:56,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:55:56,796 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['276563 memcached\n'], []) 2021-12-16 10:55:56,796 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:55:56,799 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:55:56,931 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:55:57,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:55:57,313 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:55:57,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:55:57,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:55:57,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:55:57,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:55:57,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:55:57,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:55:57,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:55:57,473 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:55:57,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:55:57,551 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['88395 memcached\n'], []) 2021-12-16 10:55:57,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:55:57,609 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:55:57,676 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['88395 memcached\n'], []) 2021-12-16 10:55:57,676 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:55:57,678 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:55:57,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:55:58,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:55:58,196 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:55:58,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:55:58,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:55:58,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:55:58,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:55:58,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:55:58,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:55:58,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:55:58,332 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:55:58,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:55:58,411 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['177568 memcached\n'], []) 2021-12-16 10:55:58,413 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:55:58,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:55:58,513 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['177568 memcached\n'], []) 2021-12-16 10:55:58,513 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:55:58,516 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:55:58,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:55:58,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:55:58,956 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:55:58,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:55:58,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:55:59,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:55:59,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:55:59,043 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:55:59,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:55:59,079 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:55:59,098 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:55:59,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:55:59,154 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['14335 memcached\n'], []) 2021-12-16 10:55:59,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:55:59,237 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:55:59,296 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['14335 memcached\n'], []) 2021-12-16 10:55:59,296 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:55:59,299 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:55:59,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:56:00,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:56:00,401 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:56:00,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:56:00,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:56:00,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:56:00,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:56:00,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:56:00,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:56:00,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:56:00,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:56:00,539 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:56:00,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:56:00,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:00,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:00,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:56:00,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:56:00,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:00,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:00,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:56:00,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:56:00,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:00,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:00,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:56:00,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:56:00,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:00,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:00,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:56:00,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:56:00,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:00,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:00,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:00,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:00,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:01,028 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:01,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:01,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:01,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:01,112 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:01,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:01,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:01,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:01,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:01,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:01,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:01,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:56:01,282 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:56:01,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:56:01,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:01,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:56:01,322 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:56:01,323 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:56:01,325 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:56:01,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:56:01,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:56:01,816 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:56:01,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:56:01,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:56:01,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:56:01,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:56:01,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:56:01,913 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:56:01,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:56:01,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:56:01,959 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:56:01,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:56:02,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:02,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:02,127 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:02,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:02,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:02,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:02,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:02,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:02,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:02,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:02,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:02,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:02,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:02,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:02,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:02,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:02,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:02,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:02,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:02,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:02,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:03,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:03,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:03,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:03,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:03,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:03,242 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:03,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:03,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:03,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:03,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:03,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:03,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:56:03,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:56:03,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,637 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:56:03,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:56:03,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:56:03,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:56:03,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:56:03,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:56:03,862 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:03,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:56:03,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:56:03,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:56:03,947 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:56:03,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:56:03,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:03,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:56:03,984 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:56:03,984 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:56:03,986 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:56:04,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:56:04,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:56:04,492 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:56:04,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:56:04,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:56:04,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:56:04,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:56:04,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:56:04,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:56:04,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:56:04,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:56:04,609 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:56:04,635 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:56:04,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:04,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:04,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:04,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:04,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:04,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:04,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:04,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:04,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:04,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:04,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:04,984 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:05,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:05,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:05,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,249 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:05,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:05,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:05,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:05,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:05,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:05,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,594 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:05,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:05,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:05,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:05,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:05,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:05,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:05,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:05,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:05,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:05,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,076 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:06,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:06,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:06,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:06,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:06,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:06,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,446 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:06,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:06,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:56:06,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:56:06,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:56:06,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:56:06,828 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:06,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:06,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:56:06,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:56:06,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:07,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:07,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:56:07,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:56:07,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:07,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:07,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:56:07,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:56:07,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:56:07,173 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:56:07,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:56:07,190 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:07,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:56:07,232 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:56:07,232 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:56:07,234 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:56:07,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:56:07,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:56:07,740 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:56:07,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:56:07,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:56:07,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:56:07,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:56:07,818 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:56:07,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:56:07,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:56:07,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:56:07,891 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:56:07,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:56:07,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:56:08,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:56:08,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,186 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:56:08,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:56:08,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:56:08,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:56:08,408 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:56:08,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:56:08,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:08,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:56:08,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:08,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:56:08,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:08,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:56:08,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:08,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:08,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:56:08,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:08,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:09,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:56:09,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:09,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:56:09,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:09,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:56:09,288 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:09,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:56:09,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:09,487 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:56:09,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:09,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:56:09,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,727 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:09,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:56:09,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:09,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:56:09,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:09,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:09,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:10,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:56:10,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:10,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:10,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:10,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:56:10,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:10,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:56:10,250 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:10,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:56:10,295 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:56:10,315 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:56:10,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:56:10,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:56:10,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:56:10,372 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:56:10,372 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:56:10,372 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:56:10,375 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:56:10,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:56:10,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:56:10,944 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:56:10,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:56:10,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:56:10,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:56:11,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:56:11,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:56:11,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:56:11,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:56:11,099 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:11,316 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:11,323 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:11,323 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:13,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:13,496 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:13,500 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:13,500 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:15,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:15,634 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:15,638 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:15,638 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:17,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:17,809 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:17,813 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:17,813 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:19,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:19,986 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:19,990 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:19,990 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:21,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:22,160 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:22,164 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:22,164 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:24,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:24,336 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:24,341 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:24,341 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:26,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:26,520 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:56:26,525 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:26,525 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:28,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:28,667 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:28,667 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:30,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:30,802 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:30,802 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:32,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:32,953 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:32,953 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:34,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:35,094 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:35,094 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:37,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:37,253 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:37,253 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:39,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:39,398 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:39,398 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:41,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:41,539 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:41,539 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:43,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:43,678 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:43,680 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:45,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:45,842 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:45,842 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:47,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:48,000 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:48,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:50,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:50,158 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:50,158 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:52,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:52,323 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:52,323 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:54,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:54,476 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:54,476 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:56,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:56,634 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:56,634 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:56:58,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:56:58,786 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:56:58,786 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:00,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:00,937 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:00,937 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:02,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:03,096 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:03,098 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:05,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:05,243 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:05,244 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:07,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:07,407 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:07,407 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:09,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:09,561 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:09,562 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:11,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:11,713 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:11,713 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:13,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:13,872 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:13,872 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:15,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:16,026 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:16,026 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:18,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:18,176 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:18,177 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:20,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:20,335 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:20,335 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:22,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:22,483 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:22,483 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:24,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:24,638 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:24,638 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:26,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:26,788 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:26,788 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:28,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:28,944 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:28,944 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:30,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:31,239 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:31,240 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:33,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:33,421 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:33,421 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:35,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:35,588 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:35,588 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:37,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:37,753 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:37,753 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:39,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:40,144 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:57:40,145 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:57:42,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:42,516 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:57:42,516 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 92.1440000534. 2021-12-16 10:57:42,517 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:57:42,519 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:57:42,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:57:42,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:57:43,005 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:57:43,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:57:43,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:57:43,045 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:57:43,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:57:43,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:57:43,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:57:43,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:57:43,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:43,328 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:57:43,328 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.811000108719. 2021-12-16 10:57:43,328 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:57:43,332 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:57:43,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:57:43,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:57:43,838 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:57:43,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:57:43,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:57:43,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:57:43,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:57:43,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:57:43,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:57:43,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:57:43,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:44,200 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:57:44,200 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871999979019. 2021-12-16 10:57:44,201 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:57:44,203 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:57:44,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:57:44,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:57:44,667 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:57:44,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:57:44,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:57:44,719 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:57:44,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:57:44,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:57:44,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:57:44,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:57:44,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:57:45,051 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:57:45,052 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.850000143051. ############################################################ # # RAM FootPrint: 3760.81640625 # ############################################################ 2021-12-16 10:57:45,082 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 10:57:45,082 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 105 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-16 10:59:30,088 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 10:59:30,088 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 10:59:35,092 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:59:35,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:35,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:59:35,747 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:35,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:59:35,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:59:35,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:59:35,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:59:35,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:59:35,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:59:35,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:59:35,855 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:59:35,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:59:35,901 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['278858 memcached\n'], []) 2021-12-16 10:59:35,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 10:59:35,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 10:59:36,059 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['278858 memcached\n'], []) 2021-12-16 10:59:36,059 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:59:36,062 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:59:36,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:36,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:59:36,489 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:36,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:59:36,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:59:36,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:59:36,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:59:36,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:59:36,556 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:59:36,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:59:36,585 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:59:36,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:59:36,628 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['89377 memcached\n'], []) 2021-12-16 10:59:36,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 10:59:36,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 10:59:36,736 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['89377 memcached\n'], []) 2021-12-16 10:59:36,736 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:59:36,740 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:59:36,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:37,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:59:37,220 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:37,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:59:37,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:59:37,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:59:37,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:59:37,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:59:37,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:59:37,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:59:37,351 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:59:37,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:59:37,440 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['178874 memcached\n'], []) 2021-12-16 10:59:37,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 10:59:37,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 10:59:37,575 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['178874 memcached\n'], []) 2021-12-16 10:59:37,575 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:59:37,578 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:59:37,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:37,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:59:38,058 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:38,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:59:38,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:59:38,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:59:38,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:59:38,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:59:38,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:59:38,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:59:38,223 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 10:59:38,223 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:59:38,308 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['15906 memcached\n'], []) 2021-12-16 10:59:38,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 10:59:38,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 10:59:38,466 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['15906 memcached\n'], []) 2021-12-16 10:59:38,466 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:59:38,469 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 10:59:38,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:39,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 10:59:39,421 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:39,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 10:59:39,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 10:59:39,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 10:59:39,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 10:59:39,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 10:59:39,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 10:59:39,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 10:59:39,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:59:39,586 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 10:59:39,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:59:39,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:39,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:39,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:59:39,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:59:39,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:39,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:39,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:59:39,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:59:39,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:39,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:39,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:59:40,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:59:40,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:40,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:59:40,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:59:40,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:40,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:40,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:40,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:40,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:40,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:40,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:40,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:40,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:40,576 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:40,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:40,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:40,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:59:40,723 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 10:59:40,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:59:40,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:40,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:59:40,775 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 10:59:40,775 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 10:59:40,778 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 10:59:40,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:41,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 10:59:41,240 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:41,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 10:59:41,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 10:59:41,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 10:59:41,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 10:59:41,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 10:59:41,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 10:59:41,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 10:59:41,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:59:41,384 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 10:59:41,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:59:41,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:41,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:41,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:41,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:41,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:41,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:41,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:41,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:41,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:41,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:41,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:41,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:41,796 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:41,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:41,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:41,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:41,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:41,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:41,976 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:42,005 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:42,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:42,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:42,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,191 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:42,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:42,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,336 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:42,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:42,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:42,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:42,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,596 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:42,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:42,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:42,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:42,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:42,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:42,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:42,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:42,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:43,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:43,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,061 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:43,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:43,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:59:43,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:59:43,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:59:43,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:59:43,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:59:43,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:59:43,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:59:43,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:59:43,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:59:43,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:59:43,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,836 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:43,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 10:59:43,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 10:59:43,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:59:43,911 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 10:59:43,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:59:43,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:43,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:59:43,980 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 10:59:43,980 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 10:59:43,982 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:59:44,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:44,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:59:44,492 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:44,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:59:44,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:59:44,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:59:44,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:59:44,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:59:44,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:59:44,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:59:44,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:59:44,637 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 10:59:44,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:59:44,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:44,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:44,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:44,778 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:44,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:44,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:44,855 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:44,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:44,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:44,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:44,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:44,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:45,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:45,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:45,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:45,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:45,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,352 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:45,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:45,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,499 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:45,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:45,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:45,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:45,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:45,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:45,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:45,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:45,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:45,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:45,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:45,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:45,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:46,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:46,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:46,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:46,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:46,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:46,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:46,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:46,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,502 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:59:46,579 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 10:59:46,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:59:46,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 10:59:46,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:59:46,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 10:59:46,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:46,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:46,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:59:46,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 10:59:47,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:47,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:47,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:59:47,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 10:59:47,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:59:47,147 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 10:59:47,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:59:47,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:47,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:59:47,214 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 10:59:47,214 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 10:59:47,217 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 10:59:47,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:47,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 10:59:47,750 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:47,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 10:59:47,772 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 10:59:47,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 10:59:47,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 10:59:47,832 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 10:59:47,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 10:59:47,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 10:59:47,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 10:59:47,892 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 10:59:47,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 10:59:47,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:47,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:59:48,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 10:59:48,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:59:48,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 10:59:48,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:59:48,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 10:59:48,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,471 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:59:48,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 10:59:48,525 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:48,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 10:59:48,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:48,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 10:59:48,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:48,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 10:59:48,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:48,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:48,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:49,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 10:59:49,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:49,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 10:59:49,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:49,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 10:59:49,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:49,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 10:59:49,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,372 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:49,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 10:59:49,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:49,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 10:59:49,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:49,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 10:59:49,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:49,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 10:59:49,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,776 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:49,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 10:59:49,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:49,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 10:59:49,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:49,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:49,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:50,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 10:59:50,029 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:50,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 10:59:50,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:50,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 10:59:50,107 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 10:59:50,118 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 10:59:50,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 10:59:50,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 10:59:50,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 10:59:50,158 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 10:59:50,158 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 10:59:50,158 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 10:59:50,161 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 10:59:50,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 10:59:50,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 10:59:50,641 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 10:59:50,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 10:59:50,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 10:59:50,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 10:59:50,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 10:59:50,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 10:59:50,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 10:59:50,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 10:59:50,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:59:50,915 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:59:50,923 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:59:50,923 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:59:52,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:59:53,096 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:59:53,101 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:59:53,101 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:59:55,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:59:55,269 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:59:55,273 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:59:55,273 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:59:57,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:59:57,444 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:59:57,447 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:59:57,447 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 10:59:59,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 10:59:59,612 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 10:59:59,617 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 10:59:59,618 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:01,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:01,801 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:00:01,805 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:01,805 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:03,806 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:03,973 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:00:03,979 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:03,979 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:05,979 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:06,158 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:00:06,164 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:06,164 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:08,165 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:08,313 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:08,313 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:10,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:10,467 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:10,467 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:12,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:12,609 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:12,609 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:14,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:14,786 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:14,786 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:16,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:16,982 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:16,982 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:18,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:19,140 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:19,140 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:21,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:21,313 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:21,313 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:23,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:23,467 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:23,467 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:25,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:25,612 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:25,614 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:27,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:27,757 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:27,757 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:29,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:29,914 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:29,915 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:31,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:32,075 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:32,075 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:34,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:34,230 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:34,232 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:36,232 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:36,394 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:36,394 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:38,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:38,553 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:38,553 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:40,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:40,709 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:40,710 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:42,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:42,861 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:42,861 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:44,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:45,019 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:45,019 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:47,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:47,187 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:47,187 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:49,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:49,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:49,349 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:51,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:51,506 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:51,507 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:53,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:53,667 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:53,667 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:55,667 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:55,829 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:55,829 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:57,829 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:00:57,986 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:00:57,987 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:00:59,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:00,141 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:00,142 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:02,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:02,308 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:02,308 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:04,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:04,466 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:04,467 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:06,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:06,687 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:06,687 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:08,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:08,848 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:08,848 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:10,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:11,006 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:11,006 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:13,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:13,176 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:13,177 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:15,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:15,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:15,348 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:17,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:17,503 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:17,505 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:19,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:19,674 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:19,674 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:01:21,676 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:21,841 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:01:21,842 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 91.6820001602. 2021-12-16 11:01:21,842 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:01:21,845 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:01:21,971 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:01:22,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:01:22,368 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:01:22,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:01:22,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:01:22,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:01:22,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:01:22,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:01:22,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:01:22,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:01:22,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:22,698 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:01:22,698 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855999946594. 2021-12-16 11:01:22,700 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:01:22,701 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:01:22,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:01:23,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:01:23,220 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:01:23,221 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:01:23,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:01:23,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:01:23,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:01:23,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:01:23,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:01:23,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:01:23,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:23,520 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:01:23,522 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.821000099182. 2021-12-16 11:01:23,522 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:01:23,523 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:01:23,599 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:01:23,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:01:23,861 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:01:23,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:01:23,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:01:23,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:01:23,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:01:23,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:01:23,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:01:23,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:01:23,963 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:24,125 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:24,125 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 11:01:26,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:26,276 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:01:26,276 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 11:01:28,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:01:28,687 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:01:28,688 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 5.16699981689. ############################################################ # # RAM FootPrint: 4399.26953125 # ############################################################ 2021-12-16 11:01:28,726 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 11:01:28,726 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 116 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-16 11:03:24,733 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 11:03:24,733 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 11:03:29,736 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:03:29,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:30,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:03:30,260 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:30,262 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:03:30,273 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:03:30,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:03:30,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:03:30,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:03:30,329 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:03:30,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:03:30,357 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:03:30,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:03:30,413 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['280558 memcached\n'], []) 2021-12-16 11:03:30,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 11:03:30,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:03:30,499 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['280558 memcached\n'], []) 2021-12-16 11:03:30,500 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:03:30,503 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:03:30,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:30,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:03:31,025 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:31,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:03:31,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:03:31,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:03:31,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:03:31,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:03:31,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:03:31,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:03:31,164 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:03:31,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:03:31,230 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['90375 memcached\n'], []) 2021-12-16 11:03:31,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 11:03:31,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:03:31,313 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['90375 memcached\n'], []) 2021-12-16 11:03:31,315 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:03:31,318 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:03:31,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:31,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:03:31,766 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:31,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:03:31,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:03:31,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:03:31,819 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:03:31,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:03:31,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:03:31,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:03:31,894 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:03:31,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:03:31,944 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['180188 memcached\n'], []) 2021-12-16 11:03:31,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 11:03:32,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:03:32,051 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['180188 memcached\n'], []) 2021-12-16 11:03:32,052 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:03:32,055 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:03:32,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:32,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:03:32,519 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:32,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:03:32,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:03:32,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:03:32,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:03:32,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:03:32,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:03:32,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:03:32,700 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:03:32,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:03:32,790 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['17519 memcached\n'], []) 2021-12-16 11:03:32,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 11:03:32,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:03:32,947 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['17519 memcached\n'], []) 2021-12-16 11:03:32,948 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:03:32,951 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:03:33,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:33,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:03:33,457 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:33,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:03:33,477 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:03:33,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:03:33,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:03:33,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:03:33,551 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:03:33,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:03:33,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:03:33,601 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 11:03:33,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:03:33,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:33,686 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:33,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:03:33,746 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:03:33,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:33,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:33,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:03:33,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:03:33,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:33,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:33,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:03:33,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:03:33,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:33,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:33,961 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:03:33,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:03:33,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:34,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:34,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:34,071 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:34,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:34,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:34,153 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:34,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:34,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:34,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:34,272 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:34,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:34,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:34,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:34,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:34,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:03:34,357 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 11:03:34,357 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:03:34,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:34,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:03:34,392 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 11:03:34,394 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:03:34,397 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:03:34,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:34,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:03:34,844 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:34,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:03:34,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:03:34,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:03:34,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:03:34,921 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:03:34,937 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:03:34,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:03:34,969 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:03:34,993 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 11:03:35,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:03:35,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,089 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:35,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:35,201 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:35,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:35,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:35,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:35,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:35,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:35,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,660 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:35,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:35,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:35,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:35,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:35,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:35,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:35,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:36,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:36,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:36,140 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:36,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:36,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:36,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:36,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,423 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:36,474 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:36,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:36,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:36,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,681 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:36,757 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:36,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,854 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:36,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:36,905 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:36,944 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:36,982 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:03:37,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:03:37,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:37,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:03:37,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:03:37,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:37,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:03:37,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:03:37,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:37,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:03:37,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:03:37,374 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:37,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:03:37,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:03:37,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:37,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:03:37,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:03:37,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:03:37,579 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 11:03:37,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:03:37,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:37,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:03:37,628 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 11:03:37,628 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:03:37,631 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:03:37,752 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:38,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:03:38,141 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:38,141 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:03:38,160 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:03:38,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:03:38,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:03:38,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:03:38,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:03:38,243 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:03:38,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:03:38,279 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 11:03:38,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:03:38,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:38,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:38,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:38,509 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:38,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,592 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:38,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:38,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:38,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:38,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:38,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:38,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:38,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:38,986 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:39,007 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:39,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:39,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:39,151 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:39,255 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:39,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:39,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:39,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:39,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:39,480 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:39,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:39,565 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,612 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:39,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:39,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,706 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:39,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:39,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,783 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:39,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:39,815 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:39,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:39,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:39,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:39,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:03:39,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:03:39,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:40,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:40,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:03:40,102 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:03:40,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:40,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:40,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:03:40,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:03:40,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:40,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:40,313 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:03:40,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:03:40,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:40,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:40,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:03:40,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:03:40,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:03:40,437 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 11:03:40,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:03:40,450 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:40,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:03:40,474 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 11:03:40,476 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:03:40,477 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:03:40,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:40,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:03:40,947 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:40,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:03:40,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:03:40,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:03:41,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:03:41,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:03:41,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:03:41,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:03:41,066 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:03:41,088 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 11:03:41,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:03:41,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:03:41,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:03:41,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:03:41,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:03:41,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:03:41,431 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:03:41,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,517 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:03:41,548 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:03:41,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,637 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:41,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:03:41,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:41,780 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:03:41,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,875 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:41,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:03:41,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:41,956 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:41,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:41,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:03:42,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:42,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:03:42,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:42,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:03:42,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:42,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:03:42,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,371 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:42,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:03:42,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,494 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:42,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:03:42,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,571 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:42,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:03:42,653 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,688 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:42,759 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:03:42,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:42,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:03:42,904 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:42,933 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:42,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:42,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:03:43,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:43,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:43,059 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:43,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:03:43,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:43,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:03:43,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:43,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:03:43,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:03:43,224 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 11:03:43,224 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:03:43,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:03:43,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:03:43,272 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 11:03:43,272 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:03:43,272 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:03:43,275 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:03:43,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:03:43,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:03:43,782 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:03:43,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:03:43,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:03:43,826 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:03:43,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:03:43,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:03:43,895 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:03:43,915 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:03:43,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:44,095 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:44,099 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:44,099 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:46,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:46,286 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:46,290 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:46,290 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:48,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:48,453 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:48,460 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:48,460 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:50,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:50,628 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:50,631 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:50,631 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:52,631 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:52,816 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:52,821 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:52,822 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:54,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:54,990 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:54,993 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:54,993 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:56,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:57,171 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:57,174 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:57,174 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:03:59,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:03:59,345 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:03:59,348 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:03:59,348 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:01,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:01,529 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:04:01,532 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:01,532 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:03,533 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:03,707 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:04:03,710 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:03,710 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:05,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:05,887 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:04:05,891 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:05,891 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:07,891 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:08,035 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:08,035 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:10,035 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:10,196 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:10,196 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:12,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:12,345 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:12,345 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:14,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:14,489 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:14,489 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:16,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:16,644 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:16,644 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:18,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:18,785 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:18,786 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:20,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:20,930 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:20,930 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:22,930 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:23,073 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:23,075 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:25,075 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:25,220 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:25,220 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:27,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:27,378 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:27,380 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:29,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:29,539 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:29,539 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:31,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:31,690 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:31,690 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:33,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:33,848 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:33,848 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:35,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:36,000 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:36,000 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:38,000 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:38,157 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:38,158 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:40,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:40,329 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:40,329 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:42,331 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:42,494 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:42,494 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:44,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:44,641 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:44,641 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:46,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:46,798 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:46,798 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:48,798 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:48,956 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:48,957 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:50,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:51,111 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:51,111 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:53,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:53,267 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:53,267 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:55,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:55,427 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:55,427 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:57,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:57,581 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:57,582 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:04:59,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:04:59,740 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:04:59,740 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:01,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:01,892 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:01,892 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:03,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:04,049 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:04,049 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:06,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:06,206 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:06,206 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:08,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:08,378 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:08,378 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:10,378 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:10,558 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:10,558 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:12,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:12,723 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:12,723 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:14,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:14,891 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:14,891 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:16,892 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:17,072 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:17,072 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:19,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:19,279 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:05:19,279 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:05:21,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:21,683 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:05:21,683 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.4110000134. 2021-12-16 11:05:21,684 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:05:21,686 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:05:21,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:05:22,124 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:05:22,217 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:05:22,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:05:22,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:05:22,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:05:22,283 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:05:22,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:05:22,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:05:22,348 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:05:22,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:22,572 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:05:22,572 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.888000011444. 2021-12-16 11:05:22,572 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:05:22,573 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:05:22,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:05:23,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:05:23,099 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:05:23,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:05:23,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:05:23,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:05:23,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:05:23,167 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:05:23,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:05:23,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:05:23,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:23,421 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:05:23,421 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.849999904633. 2021-12-16 11:05:23,423 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:05:23,424 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:05:23,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:05:23,874 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:05:24,062 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:05:24,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:05:24,130 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:05:24,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:05:24,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:05:24,219 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:05:24,321 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:05:24,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:05:24,426 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:05:24,648 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:05:24,648 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.22599983215. ############################################################ # # RAM FootPrint: 4227.5 # ############################################################ 2021-12-16 11:05:24,680 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 11:05:24,680 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 90 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-16 11:06:54,684 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 11:06:54,684 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 11:06:59,687 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:06:59,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:00,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:07:00,496 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:00,496 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:07:00,604 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:07:00,624 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:07:00,647 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:07:00,670 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:07:00,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:07:00,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:07:00,720 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:07:00,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:07:00,802 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['282735 memcached\n'], []) 2021-12-16 11:07:00,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 11:07:00,887 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:07:00,961 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['282735 memcached\n'], []) 2021-12-16 11:07:00,961 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:07:00,964 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:07:01,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:01,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:07:01,459 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:01,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:07:01,476 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:07:01,492 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:07:01,510 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:07:01,530 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:07:01,545 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:07:01,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:07:01,579 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:07:01,581 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:07:01,638 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['91420 memcached\n'], []) 2021-12-16 11:07:01,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 11:07:01,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:07:01,757 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['91420 memcached\n'], []) 2021-12-16 11:07:01,759 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:07:01,762 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:07:01,881 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:02,187 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:07:02,278 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:02,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:07:02,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:07:02,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:07:02,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:07:02,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:07:02,369 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:07:02,382 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:07:02,397 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:07:02,397 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:07:02,451 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['181530 memcached\n'], []) 2021-12-16 11:07:02,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 11:07:02,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:07:02,540 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['181530 memcached\n'], []) 2021-12-16 11:07:02,542 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:07:02,543 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:07:02,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:02,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:07:03,003 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:03,003 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:07:03,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:07:03,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:07:03,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:07:03,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:07:03,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:07:03,127 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:07:03,144 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:07:03,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:07:03,220 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['19168 memcached\n'], []) 2021-12-16 11:07:03,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 11:07:03,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:07:03,365 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['19168 memcached\n'], []) 2021-12-16 11:07:03,365 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:07:03,368 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:07:03,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:07,954 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:07:08,039 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:08,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:07:08,058 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:07:08,073 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:07:08,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:07:08,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:07:08,131 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:07:08,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:07:08,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:07:08,180 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 11:07:08,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:07:08,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,269 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:07:08,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:07:08,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:07:08,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:07:08,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,566 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:07:08,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:07:08,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,637 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:07:08,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:07:08,721 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,799 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:08,825 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:08,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,897 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:08,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:08,934 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:08,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:08,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:09,019 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:09,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:09,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:09,088 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:09,114 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:09,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:09,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:09,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:09,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:09,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:07:09,220 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 11:07:09,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:07:09,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:09,247 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:07:09,259 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 11:07:09,259 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:07:09,263 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:07:09,388 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:09,644 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:07:09,736 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:09,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:07:09,762 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:07:09,785 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:07:09,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:07:09,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:07:09,858 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:07:09,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:07:09,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:07:09,917 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 11:07:09,946 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:07:09,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,039 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:10,068 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:10,091 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:10,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:10,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:10,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:10,312 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:10,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:10,428 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:10,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:10,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,589 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:10,640 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:10,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:10,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:10,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,765 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:10,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:10,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:10,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:10,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:10,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:10,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:10,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:10,997 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,022 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:11,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:11,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,122 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:11,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:11,196 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:11,285 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:11,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,359 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:11,391 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:11,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,438 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:07:11,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:07:11,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:07:11,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:07:11,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:07:11,671 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:07:11,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:07:11,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:07:11,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:07:11,867 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:07:11,885 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:11,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:11,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:07:11,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:07:11,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:07:12,009 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 11:07:12,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:07:12,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:12,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:07:12,085 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 11:07:12,085 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:07:12,091 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:07:12,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:12,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:07:12,605 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:12,605 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:07:12,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:07:12,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:07:12,673 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:07:12,698 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:07:12,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:07:12,730 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:07:12,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:07:12,769 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 11:07:12,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:07:12,865 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:12,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:12,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:12,948 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:12,977 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:13,072 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:13,098 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,134 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:13,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:13,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,259 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:13,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:13,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:13,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:13,457 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,483 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:13,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:13,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,611 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:13,634 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:13,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:13,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:13,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:13,851 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:13,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:13,911 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:13,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:13,980 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:14,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:14,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:14,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,206 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:14,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:14,323 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,400 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:14,430 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:14,459 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,497 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,540 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:14,563 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:14,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,658 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:14,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:14,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,760 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:07:14,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:07:14,816 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,880 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:07:14,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:07:14,924 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:14,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:14,987 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:07:15,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:07:15,038 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:15,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:15,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:07:15,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:07:15,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:15,204 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:15,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:07:15,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:07:15,275 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:07:15,289 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 11:07:15,289 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:07:15,302 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:15,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:07:15,334 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 11:07:15,335 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:07:15,338 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:07:15,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:15,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:07:15,753 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:15,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:07:15,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:07:15,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:07:15,793 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:07:15,809 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:07:15,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:07:15,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:07:15,848 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:07:15,864 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 11:07:15,888 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:07:15,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:15,926 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:15,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:07:15,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:07:15,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:07:16,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:07:16,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,104 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:07:16,145 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:07:16,164 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:07:16,227 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:07:16,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,267 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:16,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:07:16,328 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,349 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:16,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:07:16,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,469 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:16,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:07:16,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:16,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:07:16,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:16,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:07:16,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,729 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,766 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:16,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:07:16,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:16,908 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:16,960 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:17,002 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:07:17,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:17,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:17,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:17,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:07:17,253 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:17,338 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:17,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:17,519 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:07:17,595 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:17,713 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:17,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:17,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:07:17,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:18,026 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:18,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:18,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:07:18,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:18,436 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:18,532 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:18,618 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:07:18,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:18,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:18,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:18,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:07:19,063 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:19,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:19,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:19,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:07:19,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:19,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:07:19,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:19,711 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:07:19,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:07:19,859 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 11:07:19,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:07:19,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:07:20,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:07:20,072 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 11:07:20,072 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:07:20,073 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:07:20,075 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:07:20,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:07:20,461 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:07:20,552 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:07:20,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:07:20,572 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:07:20,588 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:07:20,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:07:20,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:07:20,641 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:07:20,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:07:20,678 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:20,841 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:07:20,844 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:20,845 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:22,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:23,006 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:07:23,010 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:23,012 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:25,012 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:25,188 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:07:25,193 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:25,193 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:27,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:27,355 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:07:27,361 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:27,361 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:29,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:29,538 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:07:29,542 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:29,542 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:31,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:31,686 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:31,686 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:33,687 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:33,839 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:33,839 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:35,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:35,992 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:35,993 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:37,993 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:38,144 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:38,144 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:40,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:40,292 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:40,293 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:42,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:42,446 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:42,446 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:44,447 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:44,632 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:44,632 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:46,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:46,799 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:46,801 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:48,801 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:48,966 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:48,966 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:50,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:51,124 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:51,124 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:53,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:53,280 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:53,282 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:55,282 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:55,447 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:55,447 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:57,448 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:57,605 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:57,605 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:07:59,607 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:07:59,763 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:07:59,763 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:01,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:01,918 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:01,918 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:03,918 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:04,082 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:04,082 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:06,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:06,239 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:06,239 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:08,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:08,398 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:08,398 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:10,398 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:10,546 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:10,546 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:12,546 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:12,701 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:12,701 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:14,701 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:14,852 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:14,852 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:16,852 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:17,009 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:17,009 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:19,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:19,168 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:19,170 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:21,170 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:21,325 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:21,325 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:23,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:23,483 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:23,483 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:25,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:25,869 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:25,871 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:27,871 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:28,032 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:28,033 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:30,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:30,200 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:30,200 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:32,200 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:32,391 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:32,392 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:34,392 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:34,569 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:34,569 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:36,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:36,740 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:36,740 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:38,740 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:38,908 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:38,908 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:40,910 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:41,092 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:08:41,092 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:08:43,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:43,322 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:08:43,322 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 83.248000145. 2021-12-16 11:08:43,322 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:08:43,323 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:08:43,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:08:43,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:08:43,841 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:08:43,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:08:43,859 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:08:43,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:08:43,900 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:08:43,923 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:08:43,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:08:43,959 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:08:43,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:44,197 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:08:44,197 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.875. 2021-12-16 11:08:44,197 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:08:44,200 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:08:44,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:08:44,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:08:44,716 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:08:44,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:08:44,736 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:08:44,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:08:44,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:08:44,805 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:08:44,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:08:44,841 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:08:44,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:45,042 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:08:45,042 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.84500002861. 2021-12-16 11:08:45,045 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:08:45,046 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:08:45,155 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:08:45,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:08:45,528 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:08:45,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:08:45,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:08:45,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:08:45,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:08:45,733 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:08:45,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:08:45,773 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:08:45,789 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:08:45,996 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:08:45,997 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.952999830246. ############################################################ # # RAM FootPrint: 4158.86328125 # ############################################################ 2021-12-16 11:08:46,032 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 11:08:46,032 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 101 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-16 11:10:27,036 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 11:10:27,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 11:10:32,039 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:10:32,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:32,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:10:32,568 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:32,568 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:10:32,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:10:32,602 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:10:32,625 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:10:32,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:10:32,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:10:32,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:10:32,694 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:10:32,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:10:32,762 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['283918 memcached\n'], []) 2021-12-16 11:10:32,763 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 11:10:32,818 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 11:10:32,888 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['283918 memcached\n'], []) 2021-12-16 11:10:32,888 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:10:32,890 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:10:33,010 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:33,315 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:10:33,405 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:33,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:10:33,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:10:33,444 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:10:33,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:10:33,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:10:33,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:10:33,535 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:10:33,561 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.67 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:10:33,561 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:10:33,645 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.67 - Process info before sending signal: (['92434 memcached\n'], []) 2021-12-16 11:10:33,645 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: kill -9 $(pgrep memcached) 2021-12-16 11:10:33,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: pgrep -l memcached 2021-12-16 11:10:33,790 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.67 - Process info after sending signal: (['92434 memcached\n'], []) 2021-12-16 11:10:33,790 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:10:33,793 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:10:33,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:34,236 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:10:34,325 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:34,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:10:34,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:10:34,362 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:10:34,384 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:10:34,404 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:10:34,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:10:34,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:10:34,454 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:10:34,456 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:10:34,528 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['182824 memcached\n'], []) 2021-12-16 11:10:34,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 11:10:34,582 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 11:10:34,628 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['182824 memcached\n'], []) 2021-12-16 11:10:34,628 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:10:34,631 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:10:34,739 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:34,996 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:10:35,096 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:35,096 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:10:35,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:10:35,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:10:35,177 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:10:35,207 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:10:35,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:10:35,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:10:35,276 | infra | DEBUG | Thread-4 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 11:10:35,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:10:35,359 | infra | DEBUG | Thread-4 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['20749 memcached\n'], []) 2021-12-16 11:10:35,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 11:10:35,441 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 11:10:35,510 | infra | DEBUG | Thread-4 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['20749 memcached\n'], []) 2021-12-16 11:10:35,510 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:10:35,513 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:10:35,632 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:36,180 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:10:36,265 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:36,265 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:10:36,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:10:36,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:10:36,325 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:10:36,346 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:10:36,364 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:10:36,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:10:36,400 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:10:36,417 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 11:10:36,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:10:36,464 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:36,505 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:36,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:10:36,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:10:36,617 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:36,655 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:36,697 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:10:36,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:10:36,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:36,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:36,818 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:10:36,845 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:10:36,869 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:36,898 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:36,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:10:36,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:10:36,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,006 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:37,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:37,082 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:37,108 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,142 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:37,184 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:37,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:37,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,270 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:37,305 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:37,332 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:37,355 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,381 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:37,405 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:37,424 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:37,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:37,484 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:37,500 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:37,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:10:37,526 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 11:10:37,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:10:37,539 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:37,553 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:10:37,565 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 11:10:37,565 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:10:37,569 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:10:37,668 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:37,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:10:38,016 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:38,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:10:38,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:10:38,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:10:38,078 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:10:38,101 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:10:38,119 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:10:38,138 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:10:38,158 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:10:38,177 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 11:10:38,203 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:10:38,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,265 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,299 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:38,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:38,344 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,400 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:38,420 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:38,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:38,512 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:38,528 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,569 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:38,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:38,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,621 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,642 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:38,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:38,674 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:38,732 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:38,747 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,767 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,788 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:38,803 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:38,821 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,839 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:38,878 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:38,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,914 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:38,938 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:38,957 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:38,974 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:38,994 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,016 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:39,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:39,046 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,069 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,095 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:39,111 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:39,125 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,178 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:39,194 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:39,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,240 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:39,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:39,290 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,308 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,339 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:39,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:39,377 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,434 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:39,454 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:39,473 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,513 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:10:39,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:10:39,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,598 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:10:39,614 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:10:39,628 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,657 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:10:39,693 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:10:39,707 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,737 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,756 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:10:39,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:10:39,790 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,811 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:39,831 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:10:39,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 11:10:39,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:10:39,884 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 11:10:39,884 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:10:39,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:39,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:10:39,936 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 11:10:39,936 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:10:39,937 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:10:40,052 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:40,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:10:40,411 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:40,411 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:10:40,427 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:10:40,443 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:10:40,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:10:40,479 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:10:40,493 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:10:40,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:10:40,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:10:40,535 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 11:10:40,562 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:10:40,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:40,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:40,661 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:40,691 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:40,714 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:40,743 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:40,769 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:40,786 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:40,802 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:40,822 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:40,844 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:40,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:40,877 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:40,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:40,927 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:40,953 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:40,973 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:40,999 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,025 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:41,043 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:41,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,085 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,117 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:41,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:41,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,176 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:41,216 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:41,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,256 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,278 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:41,296 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:41,322 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,390 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:41,417 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:41,437 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,470 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,503 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:41,529 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:41,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,575 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:41,630 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:41,648 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,684 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,720 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:41,753 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:41,782 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,838 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:41,861 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:41,890 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:41,920 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:41,947 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:41,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:41,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,017 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:42,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:42,081 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,105 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,128 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:42,154 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:42,173 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,197 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,220 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:10:42,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 11:10:42,257 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,286 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,309 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:10:42,326 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 11:10:42,345 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,375 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,410 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:10:42,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 11:10:42,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,482 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:42,507 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:10:42,526 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 11:10:42,543 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:10:42,559 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 11:10:42,559 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:10:42,573 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:42,591 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:10:42,605 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 11:10:42,605 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:10:42,608 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:10:42,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:43,040 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:10:43,115 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:43,115 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:10:43,132 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:10:43,150 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:10:43,168 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:10:43,193 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:10:43,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:10:43,226 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:10:43,244 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 11:10:43,266 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 11:10:43,293 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 11:10:43,318 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:43,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:43,414 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:10:43,453 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 11:10:43,489 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:43,536 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:43,585 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:10:43,622 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 11:10:43,654 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:43,694 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:43,726 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:10:43,755 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 11:10:43,780 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:43,813 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:43,849 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:10:43,872 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 11:10:43,894 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:43,917 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:43,950 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:43,967 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 11:10:43,983 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,013 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,033 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:44,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 11:10:44,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,118 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:44,135 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 11:10:44,157 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,181 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,209 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:44,229 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 11:10:44,252 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,280 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,311 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:44,335 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 11:10:44,361 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,394 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,433 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:44,460 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 11:10:44,486 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,516 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,555 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:44,586 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 11:10:44,615 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,683 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:44,710 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 11:10:44,734 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,770 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,812 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:44,842 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 11:10:44,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:44,901 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:44,936 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:44,964 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 11:10:44,990 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,023 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,062 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:45,086 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 11:10:45,109 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,144 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,183 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:45,213 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 11:10:45,239 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,276 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,316 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:45,342 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 11:10:45,367 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,401 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,440 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:45,466 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 11:10:45,490 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,520 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,558 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:45,578 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 11:10:45,601 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,619 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 11:10:45,638 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:45,651 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 11:10:45,664 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 11:10:45,677 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 11:10:45,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 11:10:45,690 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 11:10:45,703 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 11:10:45,714 | test | DEBUG | Thread-4 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 11:10:45,716 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:10:45,716 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:10:45,719 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 11:10:45,835 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:10:46,130 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 11:10:46,217 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:10:46,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 11:10:46,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 11:10:46,266 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 11:10:46,292 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 11:10:46,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 11:10:46,341 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 11:10:46,358 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 11:10:46,380 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:46,535 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:46,542 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:46,542 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:48,542 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:48,711 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:48,721 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:48,721 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:50,723 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:50,858 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:50,862 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:50,862 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:52,864 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:53,032 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:53,036 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:53,036 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:55,036 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:55,207 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:55,211 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:55,211 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:57,211 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:57,381 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:57,385 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:57,387 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:10:59,387 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:10:59,548 | test | WARNING | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 11:10:59,552 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:10:59,552 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:01,552 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:01,698 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:01,700 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:03,700 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:03,857 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:03,857 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:05,857 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:06,009 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:06,009 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:08,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:08,161 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:08,161 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:10,163 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:10,319 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:10,319 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:12,319 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:12,466 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:12,467 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:14,467 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:14,607 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:14,608 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:16,608 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:16,750 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:16,750 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:18,750 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:18,901 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:18,903 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:20,903 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:21,055 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:21,055 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:23,055 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:23,216 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:23,216 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:25,217 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:25,368 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:25,368 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:27,368 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:27,523 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:27,523 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:29,523 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:29,678 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:29,680 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:31,680 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:31,832 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:31,832 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:33,834 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:33,990 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:33,990 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:35,992 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:36,145 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:36,147 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:38,147 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:38,299 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:38,301 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:40,301 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:40,461 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:40,463 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:42,463 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:42,625 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:42,627 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:44,627 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:44,779 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:44,779 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:46,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:46,940 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:46,940 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:48,940 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:49,091 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:49,092 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:51,092 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:51,263 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:51,263 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:53,263 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:53,421 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:53,421 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:55,421 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:55,584 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:55,584 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:11:57,584 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:11:58,051 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:11:58,051 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:00,051 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:00,506 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:00,506 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:02,506 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:02,677 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:02,677 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:04,677 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:04,845 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:04,845 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:06,846 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:07,013 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:07,013 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:09,015 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:09,174 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:09,174 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:11,174 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:11,351 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 11:12:11,351 | test | INFO | Thread-4 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 11:12:13,351 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:13,631 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 11:12:13,631 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 87.9149999619. 2021-12-16 11:12:13,631 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:12:13,634 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 11:12:13,749 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:12:14,056 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 11:12:14,148 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:12:14,148 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 11:12:14,171 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 11:12:14,188 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 11:12:14,210 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 11:12:14,230 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 11:12:14,246 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 11:12:14,260 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 11:12:14,279 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:14,434 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 11:12:14,434 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.802999973297. 2021-12-16 11:12:14,434 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:12:14,437 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 11:12:14,549 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:12:14,868 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 11:12:14,951 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:12:14,951 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 11:12:14,970 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 11:12:14,989 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 11:12:15,009 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 11:12:15,032 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 11:12:15,049 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 11:12:15,065 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 11:12:15,084 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:15,262 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 11:12:15,262 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.827000141144. 2021-12-16 11:12:15,262 | test | DEBUG | Thread-4 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 11:12:15,265 | infra | DEBUG | Thread-4 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 11:12:15,407 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 11:12:15,650 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 11:12:15,717 | infra | DEBUG | Thread-4 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 11:12:15,717 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 11:12:15,779 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 11:12:15,907 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 11:12:16,007 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 11:12:16,042 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 11:12:16,137 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 11:12:16,161 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 11:12:16,233 | infra | DEBUG | Thread-4 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 11:12:16,450 | infra | DEBUG | Thread-4 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 11:12:16,450 | test | INFO | Thread-4 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.18799996376. 2021-12-16 11:12:16,456 | test | DEBUG | Thread-4 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-16 11:12:16,456 | test | DEBUG | Thread-4 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 11: Failover a node and RebalanceOut that node with loading in parallel # ############################################################ 2021-12-16 11:17:16,604 | test | DEBUG | MainThread | [rest_client:fail_over:1374] ns_1@172.23.110.67 - Failover successful 2021-12-16 11:17:16,611 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L577 2021-12-16 11:17:16,611 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 30 seconds. Reason: None 2021-12-16 11:17:46,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 11:17:46,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 11:17:51,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:17:51,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:17:54,663 | test | INFO | MainThread | [rest_client:monitorRebalance:1584] Rebalance done. Taken 8.05200004578 seconds to complete 2021-12-16 11:17:54,671 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1587 2021-12-16 11:17:54,671 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 8.05200004578 seconds. Reason: Wait after rebalance complete 2021-12-16 11:18:02,749 | test | DEBUG | MainThread | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.67,ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': u'ns_1@172.23.110.67', 'user': 'Administrator'} 2021-12-16 11:18:02,801 | test | DEBUG | MainThread | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 11:18:02,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:02,816 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:02,816 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:05,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:05,836 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:05,836 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:08,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:08,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:08,855 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:11,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:11,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:11,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:14,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:14,897 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:14,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:17,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:17,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:17,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:20,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:20,950 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:20,950 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:23,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:23,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:23,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:26,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:26,987 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:26,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:30,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:30,009 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:30,009 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:33,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:33,028 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:33,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:36,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:36,046 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:36,046 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:39,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:39,066 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:39,066 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:42,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:42,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:42,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:45,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:45,112 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:45,114 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:48,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:48,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:48,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:51,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:51,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:51,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:54,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:54,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:54,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:18:57,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:18:57,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:18:57,201 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:00,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:19:00,226 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:00,226 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:03,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:19:03,247 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:03,247 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:06,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:19:06,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:06,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:09,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:19:09,289 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:09,289 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:12,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 11:19:12,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:12,306 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:15,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-16 11:19:15,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:15,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:18,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-16 11:19:18,362 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:18,362 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:21,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-16 11:19:21,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:21,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:24,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-16 11:19:24,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:24,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:27,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.20 % 2021-12-16 11:19:27,424 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:27,424 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:30,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-16 11:19:30,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:30,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:33,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.39 % 2021-12-16 11:19:33,477 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:33,477 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:36,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-16 11:19:36,523 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:36,523 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:39,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.59 % 2021-12-16 11:19:39,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:39,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:42,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:42,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:42,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:45,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:45,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:45,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:48,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:48,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:48,599 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:51,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:51,627 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:51,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:54,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:54,647 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:54,647 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:19:57,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:19:57,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:19:57,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:00,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:00,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:00,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:03,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:03,704 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:03,704 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:06,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:06,724 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:06,724 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:09,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:09,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:09,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:12,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:12,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:12,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:15,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:15,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:15,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:18,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:18,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:18,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:21,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:21,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:21,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:24,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:24,854 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:24,854 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:27,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:27,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:27,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:30,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:30,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:30,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:33,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:33,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:33,908 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:36,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:36,928 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:36,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:39,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:39,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:39,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:42,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.78 % 2021-12-16 11:20:42,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:42,971 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:45,984 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.97 % 2021-12-16 11:20:45,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:45,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:49,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.17 % 2021-12-16 11:20:49,007 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:49,009 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:52,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-16 11:20:52,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:52,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:55,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.37 % 2021-12-16 11:20:55,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:55,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:20:58,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:20:58,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:20:58,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:01,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:01,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:01,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:04,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:04,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:04,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:07,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:07,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:07,137 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:10,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:10,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:10,158 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:13,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:13,183 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:13,183 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:16,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:16,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:16,201 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:19,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:19,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:19,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:22,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:22,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:22,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:25,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:25,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:25,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:28,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:28,280 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:28,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:31,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:31,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:31,306 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:34,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:34,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:34,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:37,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:37,344 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:37,344 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:40,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:40,364 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:40,364 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:43,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:43,387 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:43,387 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:46,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:46,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:46,405 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:49,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:49,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:49,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:52,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:52,448 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:52,448 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:55,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:55,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:55,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:21:58,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:21:58,484 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:21:58,484 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:01,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:22:01,505 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:01,505 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:04,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:22:04,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:04,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:07,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.56 % 2021-12-16 11:22:07,543 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:07,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:10,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-16 11:22:10,561 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:10,561 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:13,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-16 11:22:13,576 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:13,576 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:16,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 1.76 % 2021-12-16 11:22:16,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:16,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:19,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:19,622 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:19,624 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:22,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:22,645 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:22,645 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:25,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:25,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:25,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:28,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:28,678 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:28,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:31,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:31,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:31,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:34,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:34,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:34,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:37,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:37,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:37,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:40,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:40,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:40,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:43,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:43,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:43,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:46,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:46,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:46,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:49,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:49,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:49,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:52,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:52,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:52,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:55,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:55,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:55,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:22:58,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:22:58,862 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:22:58,862 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:01,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:01,880 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:01,880 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:04,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:04,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:04,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:07,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:07,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:07,913 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:10,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:10,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:10,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:13,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:13,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:13,969 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:16,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:16,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:16,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:20,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.15 % 2021-12-16 11:23:20,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:20,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:23,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-16 11:23:23,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:23,020 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:26,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.34 % 2021-12-16 11:23:26,045 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:26,045 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:29,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-16 11:23:29,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:29,063 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:32,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-16 11:23:32,079 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:32,081 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:35,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-16 11:23:35,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:35,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:38,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-16 11:23:38,114 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:38,114 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:41,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.54 % 2021-12-16 11:23:41,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:41,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:44,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:44,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:44,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:47,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:47,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:47,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:50,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:50,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:50,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:53,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:53,211 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:53,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:56,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:56,229 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:56,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:23:59,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:23:59,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:23:59,259 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:02,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:02,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:02,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:05,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:05,295 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:05,295 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:08,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:08,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:08,312 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:11,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:11,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:11,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:14,338 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:14,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:14,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:17,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:17,364 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:17,364 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:20,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:20,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:20,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:23,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:23,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:23,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:26,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:26,423 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:26,423 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:29,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:29,438 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:29,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:32,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.73 % 2021-12-16 11:24:32,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:32,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:35,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-16 11:24:35,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:35,476 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:38,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-16 11:24:38,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:38,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:41,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-16 11:24:41,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:41,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:44,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 2.93 % 2021-12-16 11:24:44,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:44,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:47,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:24:47,548 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:47,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:50,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:24:50,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:50,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:53,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:24:53,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:53,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:56,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:24:56,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:56,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:24:59,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:24:59,625 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:24:59,625 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:02,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.12 % 2021-12-16 11:25:02,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:02,653 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:05,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:05,673 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:05,673 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:08,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:08,691 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:08,691 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:11,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:11,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:11,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:14,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:14,729 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:14,730 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:17,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:17,744 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:17,744 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:20,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:20,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:20,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:23,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:23,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:23,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:26,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:26,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:26,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:29,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:29,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:29,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:32,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:32,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:32,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:35,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:35,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:35,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:38,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:38,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:38,913 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:41,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:41,933 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:41,933 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:44,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:44,948 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:44,948 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:47,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:47,966 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:47,966 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:50,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:50,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:50,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:53,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.32 % 2021-12-16 11:25:54,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:54,002 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:25:57,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.51 % 2021-12-16 11:25:57,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:25:57,022 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:00,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.51 % 2021-12-16 11:26:00,040 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:00,040 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:03,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.51 % 2021-12-16 11:26:03,058 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:03,058 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:06,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.51 % 2021-12-16 11:26:06,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:06,073 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:09,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.51 % 2021-12-16 11:26:09,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:09,091 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:12,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 11:26:12,109 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:12,109 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:15,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.71 % 2021-12-16 11:26:15,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:15,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:18,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:18,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:18,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:21,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:21,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:21,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:24,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:24,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:24,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:27,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:27,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:27,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:30,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:30,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:30,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:33,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:33,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:33,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:36,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:36,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:36,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:39,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:39,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:39,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:42,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:42,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:42,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:45,306 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:45,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:45,312 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:48,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:48,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:48,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:51,344 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:51,348 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:51,348 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:54,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:54,364 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:54,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:26:57,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:26:57,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:26:57,380 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:00,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:27:00,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:00,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:03,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:27:03,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:03,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:06,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:27:06,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:06,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:09,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 3.90 % 2021-12-16 11:27:09,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:09,454 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:12,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:12,471 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:12,471 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:15,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:15,487 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:15,487 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:18,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:18,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:18,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:21,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:21,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:21,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:24,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:24,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:24,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:27,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:27,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:27,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:30,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.10 % 2021-12-16 11:27:30,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:30,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:33,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.29 % 2021-12-16 11:27:33,602 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:33,602 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:36,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:36,624 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:36,624 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:39,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:39,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:39,640 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:42,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:42,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:42,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:45,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:45,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:45,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:48,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:48,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:48,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:51,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:51,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:51,714 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:54,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:54,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:54,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:27:57,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:27:57,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:27:57,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:00,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:00,779 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:00,779 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:03,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:03,795 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:03,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:06,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:06,813 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:06,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:09,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:09,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:09,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:12,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:12,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:12,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:15,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:15,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:15,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:18,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:18,880 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:18,880 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:21,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.49 % 2021-12-16 11:28:21,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:21,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:24,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:24,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:24,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:27,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:27,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:27,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:30,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:30,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:30,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:33,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:33,997 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:33,997 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:37,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:37,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:37,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:40,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:40,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:40,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:43,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:43,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:43,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:46,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:46,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:46,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:49,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:49,105 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:49,105 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:52,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:52,124 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:52,124 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:55,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:55,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:55,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:28:58,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:28:58,168 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:28:58,170 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:01,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 4.68 % 2021-12-16 11:29:01,186 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:01,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:04,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:04,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:04,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:07,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:07,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:07,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:10,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:10,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:10,243 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:13,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:16,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:16,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:19,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:19,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:19,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:22,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:22,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:22,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:25,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:25,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:25,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:28,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:28,364 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:28,364 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:31,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:31,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:31,380 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:34,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:34,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:34,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:37,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:37,415 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:37,415 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:40,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.07 % 2021-12-16 11:29:40,433 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:40,433 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:43,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:43,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:43,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:46,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:46,482 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:46,482 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:49,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:49,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:49,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:52,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:52,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:52,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:55,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:55,545 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:55,545 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:29:58,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:29:58,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:29:58,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:01,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:01,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:01,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:04,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:04,602 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:04,602 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:07,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:07,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:07,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:10,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:10,644 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:10,644 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:13,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:13,677 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:13,678 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:16,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:16,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:16,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:19,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.27 % 2021-12-16 11:30:19,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:19,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:22,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.46 % 2021-12-16 11:30:22,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:22,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:25,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:25,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:25,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:28,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:28,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:28,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:31,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:31,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:31,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:34,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:34,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:34,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:37,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:37,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:37,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:40,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:40,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:40,852 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:43,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:43,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:43,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:46,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:46,892 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:46,892 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:49,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:49,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:49,913 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:52,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:52,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:52,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:55,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:55,954 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:55,954 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:30:58,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:30:58,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:30:58,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:01,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.66 % 2021-12-16 11:31:01,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:01,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:04,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:05,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:05,006 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:08,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:08,053 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:08,053 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:11,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:11,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:11,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:14,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:14,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:14,085 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:17,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:17,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:17,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:20,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:20,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:20,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:23,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:23,140 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:23,140 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:26,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:26,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:26,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:29,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:29,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:29,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:32,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:32,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:32,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:35,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:35,210 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:35,210 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:38,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:38,227 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:38,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:41,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 5.85 % 2021-12-16 11:31:41,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:41,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:44,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:44,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:44,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:47,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:47,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:47,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:50,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:50,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:50,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:53,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:53,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:53,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:56,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:56,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:56,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:31:59,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:31:59,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:31:59,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:02,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:02,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:02,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:05,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:05,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:05,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:08,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:08,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:08,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:11,418 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:11,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:11,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:14,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:14,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:14,450 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:17,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:17,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:17,469 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:20,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.24 % 2021-12-16 11:32:20,484 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:20,484 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:23,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:23,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:23,502 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:26,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:26,523 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:26,523 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:29,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:29,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:29,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:32,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:32,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:32,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:35,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:35,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:35,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:38,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:38,594 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:38,594 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:41,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:41,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:41,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:44,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:44,627 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:44,627 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:47,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:47,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:47,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:50,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:50,668 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:50,668 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:53,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:53,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:53,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:56,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.44 % 2021-12-16 11:32:56,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:56,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:32:59,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.63 % 2021-12-16 11:32:59,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:32:59,721 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:02,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:02,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:02,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:05,778 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:05,801 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:05,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:08,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:08,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:08,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:11,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:11,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:11,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:14,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:14,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:14,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:17,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:17,872 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:17,874 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:20,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:20,892 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:20,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:23,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:23,910 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:23,910 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:26,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:26,926 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:26,926 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:29,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:29,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:29,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:32,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:32,957 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:32,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:35,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:35,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:35,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:38,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:38,992 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:38,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:42,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:42,009 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:42,009 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:45,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:45,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:45,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:48,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 6.83 % 2021-12-16 11:33:48,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:48,043 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:51,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:33:51,061 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:51,061 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:54,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:33:54,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:54,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:33:57,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:33:57,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:33:57,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:00,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:00,119 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:00,119 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:03,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:03,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:03,137 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:06,147 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:06,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:06,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:09,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:09,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:09,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:12,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:12,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:12,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:15,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:15,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:15,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:18,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:18,240 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:18,240 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:21,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.02 % 2021-12-16 11:34:21,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:21,260 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:24,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:24,278 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:24,278 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:27,288 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:27,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:27,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:30,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:30,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:30,312 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:33,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:33,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:33,331 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:36,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:36,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:36,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:39,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:39,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:39,368 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:42,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:42,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:42,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:45,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:45,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:45,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:48,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:48,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:48,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:51,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:51,443 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:51,443 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:54,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:54,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:54,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:34:57,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:34:57,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:34:57,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:00,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:00,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:00,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:03,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:03,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:03,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:06,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:06,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:06,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:09,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:09,556 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:09,556 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:12,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:12,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:12,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:15,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.41 % 2021-12-16 11:35:15,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:15,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:18,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:18,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:18,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:21,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:21,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:21,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:24,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:24,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:24,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:27,668 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:27,677 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:27,677 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:30,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:30,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:30,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:33,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:33,714 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:33,716 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:36,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:36,732 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:36,732 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:39,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:39,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:39,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:42,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.61 % 2021-12-16 11:35:42,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:42,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:45,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 11:35:45,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:45,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:48,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 7.80 % 2021-12-16 11:35:48,812 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:48,813 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:51,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:35:51,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:51,829 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:54,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:35:54,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:54,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:35:57,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:35:57,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:35:57,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:00,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:00,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:00,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:03,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:03,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:03,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:06,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:06,923 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:06,923 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:09,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:09,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:09,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:12,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:12,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:12,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:15,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:15,971 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:15,971 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:18,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:18,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:18,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:22,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.00 % 2021-12-16 11:36:22,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:22,026 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:25,038 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:25,043 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:25,043 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:28,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:28,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:28,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:31,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:31,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:31,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:34,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:34,094 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:34,094 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:37,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:37,112 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:37,112 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:40,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:40,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:40,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:43,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:43,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:43,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:46,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:46,160 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:46,160 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:49,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:49,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:49,201 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:52,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:52,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:52,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:55,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:55,240 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:55,240 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:36:58,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:36:58,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:36:58,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:01,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.19 % 2021-12-16 11:37:01,272 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:01,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:04,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:04,290 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:04,290 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:07,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:07,308 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:07,308 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:10,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:10,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:10,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:13,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:13,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:13,339 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:16,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:16,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:16,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:19,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:19,374 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:19,374 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:22,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:22,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:22,392 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:25,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:25,411 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:25,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:28,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:28,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:28,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:31,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.39 % 2021-12-16 11:37:31,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:31,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:34,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.58 % 2021-12-16 11:37:34,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:34,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:37,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.58 % 2021-12-16 11:37:37,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:37,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:40,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.58 % 2021-12-16 11:37:40,505 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:40,505 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:43,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.58 % 2021-12-16 11:37:43,522 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:43,522 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:46,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:37:46,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:46,536 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:49,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:37:49,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:49,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:52,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:37:52,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:52,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:55,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:37:55,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:55,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:37:58,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:37:58,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:37:58,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:01,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:01,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:01,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:04,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:04,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:04,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:07,681 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:07,690 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:07,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:10,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:10,711 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:10,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:13,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:13,729 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:13,729 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:16,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:16,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:16,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:19,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:19,760 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:19,762 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:22,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:22,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:22,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:25,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:25,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:25,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:28,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:28,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:28,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:31,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.78 % 2021-12-16 11:38:31,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:31,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:34,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.97 % 2021-12-16 11:38:34,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:34,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:37,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 8.97 % 2021-12-16 11:38:37,878 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:37,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:40,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:40,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:40,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:43,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:43,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:43,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:46,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:46,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:46,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:49,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:49,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:49,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:52,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:52,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:52,969 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:55,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:55,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:55,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:38:59,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:38:59,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:38:59,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:02,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:39:02,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:02,033 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:05,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:39:05,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:05,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:08,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:39:08,063 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:08,063 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:11,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:39:11,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:11,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:14,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.17 % 2021-12-16 11:39:14,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:14,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:17,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:17,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:17,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:20,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:20,135 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:20,135 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:23,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:23,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:23,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:26,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:26,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:26,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:29,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:29,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:29,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:32,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:32,198 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:32,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:35,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:35,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:35,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:38,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:38,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:38,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:41,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:41,244 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:41,244 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:44,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:44,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:44,265 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:47,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.37 % 2021-12-16 11:39:47,288 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:47,289 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:50,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:39:50,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:50,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:53,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:39:53,322 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:53,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:56,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:39:56,336 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:56,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:39:59,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:39:59,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:39:59,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:02,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:40:02,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:02,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:05,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:40:05,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:05,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:08,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.56 % 2021-12-16 11:40:08,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:08,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:11,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:11,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:11,423 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:14,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:14,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:14,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:17,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:17,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:17,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:20,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:20,479 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:20,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:23,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:23,494 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:23,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:26,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:26,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:26,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:29,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:29,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:29,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:32,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:32,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:32,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:35,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:35,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:35,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:38,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:38,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:38,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:41,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:41,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:41,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:44,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:44,617 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:44,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:47,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:47,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:47,642 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:50,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:50,663 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:50,663 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:53,676 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.76 % 2021-12-16 11:40:53,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:53,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:56,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:40:56,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:56,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:40:59,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:40:59,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:40:59,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:02,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:02,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:02,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:05,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:05,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:05,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:08,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:08,775 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:08,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:11,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:11,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:11,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:14,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:14,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:14,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:17,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:17,826 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:17,826 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:20,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:20,852 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:20,852 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:23,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 9.95 % 2021-12-16 11:41:23,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:23,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:26,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-16 11:41:26,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:26,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:29,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.15 % 2021-12-16 11:41:29,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:29,910 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:32,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:32,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:32,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:35,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:35,944 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:35,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:38,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:38,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:38,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:41,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:41,997 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:41,997 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:45,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:45,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:45,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:48,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:48,028 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:48,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:51,038 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:51,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:51,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:54,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:54,063 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:54,063 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:41:57,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:41:57,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:41:57,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:00,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:00,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:00,105 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:03,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:03,119 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:03,119 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:06,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:06,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:06,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:09,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:09,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:09,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:12,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:12,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:12,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:15,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.34 % 2021-12-16 11:42:15,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:15,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:18,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:18,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:18,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:21,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:21,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:21,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:24,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:24,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:24,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:27,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:27,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:27,280 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:30,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:30,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:30,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:33,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:33,321 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:33,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:36,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:36,336 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:36,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:39,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:39,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:39,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:42,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.54 % 2021-12-16 11:42:42,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:42,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:45,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:42:45,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:45,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:48,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:42:48,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:48,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:51,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:42:51,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:51,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:54,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:42:54,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:54,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:42:57,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:42:57,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:42:57,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:00,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:00,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:00,476 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:03,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:03,492 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:03,492 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:06,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:06,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:06,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:09,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:09,528 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:09,528 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:12,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:12,548 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:12,548 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:15,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:15,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:15,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:18,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:18,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:18,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:21,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:21,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:21,599 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:24,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:24,614 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:24,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:27,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:27,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:27,632 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:30,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:30,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:30,653 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:33,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:33,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:33,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:36,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:36,691 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:36,693 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:39,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:39,709 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:39,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:42,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.73 % 2021-12-16 11:43:42,724 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:42,724 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:45,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 10.93 % 2021-12-16 11:43:45,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:45,744 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:48,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:43:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:51,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:43:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:54,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:43:54,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:54,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:43:57,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:43:57,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:43:57,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:00,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:44:00,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:00,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:03,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.12 % 2021-12-16 11:44:03,854 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:03,855 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:06,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:06,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:06,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:09,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:09,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:09,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:12,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:12,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:12,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:15,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:15,923 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:15,923 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:18,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:18,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:18,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:21,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:21,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:21,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:24,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:24,982 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:24,982 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:27,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:27,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:27,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:31,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:31,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:31,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:34,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:34,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:34,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:37,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:37,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:37,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:40,073 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:40,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:40,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:43,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:43,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:43,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:46,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:46,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:46,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:49,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:49,127 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:49,127 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:52,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:52,142 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:52,142 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:55,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:55,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:55,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:44:58,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:44:58,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:44:58,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:01,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.32 % 2021-12-16 11:45:01,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:01,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:04,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.51 % 2021-12-16 11:45:04,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:04,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:07,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.51 % 2021-12-16 11:45:07,232 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:07,232 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:10,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.51 % 2021-12-16 11:45:10,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:10,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:13,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.51 % 2021-12-16 11:45:13,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:13,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.51 % 2021-12-16 11:45:16,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:16,285 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:19,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.71 % 2021-12-16 11:45:19,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:19,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:22,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.71 % 2021-12-16 11:45:22,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:22,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:25,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.71 % 2021-12-16 11:45:25,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:25,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:28,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:28,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:28,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:31,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:31,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:31,394 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:34,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:34,410 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:34,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:37,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:37,426 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:37,426 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:40,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:40,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:40,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:43,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:43,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:43,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:46,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:46,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:46,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:49,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:49,499 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:49,499 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:52,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:52,513 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:52,513 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:55,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:55,528 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:55,528 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:45:58,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:45:58,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:45:58,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:01,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:01,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:01,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:04,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:04,581 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:04,581 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:07,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:07,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:07,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:10,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:10,612 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:10,612 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:13,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:13,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:13,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:16,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 11.90 % 2021-12-16 11:46:16,663 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:16,663 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:19,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:19,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:19,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:22,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:22,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:22,706 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:25,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:25,721 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:25,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:28,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:28,737 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:28,737 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:31,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:31,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:31,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:34,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:34,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:34,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:37,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:37,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:37,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:40,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:40,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:40,816 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:43,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:43,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:43,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:46,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.10 % 2021-12-16 11:46:46,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:46,849 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:49,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 11:46:49,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:49,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:52,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 11:46:52,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:52,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:55,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 11:46:55,910 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:55,910 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:46:58,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.29 % 2021-12-16 11:46:58,926 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:46:58,926 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:01,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:01,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:01,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:04,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:04,957 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:04,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:07,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:07,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:07,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:10,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:10,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:10,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:14,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:14,007 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:14,007 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:17,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:17,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:17,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:20,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:20,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:20,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:23,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:23,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:23,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:26,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:26,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:26,079 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:29,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:29,094 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:29,094 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:32,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:32,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:32,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:35,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.49 % 2021-12-16 11:47:35,127 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:35,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:38,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:38,147 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:38,147 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:41,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:41,168 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:41,170 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:44,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:47,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:47,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:47,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:50,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:53,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:53,237 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:53,237 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:56,247 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:56,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:56,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:47:59,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:47:59,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:47:59,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:02,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:48:02,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:02,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:05,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:48:05,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:05,303 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:08,313 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:48:08,321 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:08,321 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:11,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.68 % 2021-12-16 11:48:11,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:11,338 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:14,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:14,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:14,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:17,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:17,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:17,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:20,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:20,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:20,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:23,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:23,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:23,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:26,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:26,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:26,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:29,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:29,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:29,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:32,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:32,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:32,490 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:35,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:35,507 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:35,507 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:38,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 12.88 % 2021-12-16 11:48:38,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:38,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:41,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:41,545 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:41,545 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:44,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:44,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:44,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:47,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:47,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:47,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:50,602 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:50,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:50,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:53,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:53,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:53,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:56,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:56,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:56,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:48:59,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.07 % 2021-12-16 11:48:59,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:48:59,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:02,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:02,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:02,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:05,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:05,730 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:05,730 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:08,740 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:08,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:08,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:11,757 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:11,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:11,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:14,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:14,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:14,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:17,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:17,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:17,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:20,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:20,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:20,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:23,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:23,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:23,852 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:26,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:26,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:26,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:29,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:29,887 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:29,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:32,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:32,911 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:32,913 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:35,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:35,936 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:35,936 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:38,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.27 % 2021-12-16 11:49:38,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:38,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:41,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:41,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:41,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:44,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:44,987 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:44,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:48,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:48,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:48,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:51,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:51,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:51,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:54,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:54,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:54,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:49:57,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:49:57,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:49:57,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:00,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:50:00,112 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:00,112 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:03,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:50:03,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:03,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:06,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:50:06,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:06,153 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:09,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.46 % 2021-12-16 11:50:09,176 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:09,176 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:12,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.66 % 2021-12-16 11:50:12,193 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:12,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:15,201 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.66 % 2021-12-16 11:50:15,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:15,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:18,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.66 % 2021-12-16 11:50:18,226 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:18,226 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:21,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:21,244 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:21,244 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:24,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:24,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:24,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:27,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:27,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:27,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:30,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:30,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:30,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:33,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:33,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:33,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:36,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:36,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:36,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:39,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:39,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:39,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:42,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:42,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:42,372 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:45,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:45,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:45,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:48,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:48,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:48,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:51,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:51,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:51,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:54,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:54,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:54,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:50:57,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:50:57,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:50:57,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:00,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:51:00,477 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:00,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:03,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 13.85 % 2021-12-16 11:51:03,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:03,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:06,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:06,507 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:06,507 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:09,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:09,525 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:09,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:12,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:12,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:12,548 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:15,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:15,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:15,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:18,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:18,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:18,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:21,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:21,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:21,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:24,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:24,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:24,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:27,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:27,635 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:27,635 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:30,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:30,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:30,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:33,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:33,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:33,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:36,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:36,690 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:36,691 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:39,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.05 % 2021-12-16 11:51:39,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:39,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:42,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.24 % 2021-12-16 11:51:42,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:42,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:45,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.24 % 2021-12-16 11:51:45,749 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:45,749 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:48,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:51:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:51,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:51:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:54,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:51:54,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:54,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:51:57,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:51:57,813 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:51:57,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:00,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:00,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:00,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:03,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:03,849 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:03,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:06,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:06,867 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:06,867 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:09,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:09,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:09,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:12,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:12,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:12,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:15,910 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:15,920 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:15,920 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:18,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:18,941 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:18,941 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:21,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:21,957 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:21,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:24,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:24,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:24,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:27,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:27,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:27,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:31,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:31,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:31,006 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:34,017 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.44 % 2021-12-16 11:52:34,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:34,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:37,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.63 % 2021-12-16 11:52:37,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:37,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:40,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.63 % 2021-12-16 11:52:40,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:40,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:43,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.63 % 2021-12-16 11:52:43,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:43,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:46,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.63 % 2021-12-16 11:52:46,105 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:46,105 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:49,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.63 % 2021-12-16 11:52:49,124 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:49,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:52,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:52:52,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:52,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:55,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:52:55,155 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:55,157 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:52:58,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:52:58,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:52:58,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:01,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:01,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:01,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:04,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:04,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:04,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:07,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:07,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:07,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:10,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:10,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:10,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:13,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:16,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:16,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:16,285 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:19,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:19,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:19,306 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:22,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:22,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:22,326 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:25,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:25,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:25,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:28,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 14.83 % 2021-12-16 11:53:28,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:28,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:31,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:31,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:31,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:34,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:34,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:34,392 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:37,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:37,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:37,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:40,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:40,433 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:40,434 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:43,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:43,448 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:43,448 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:46,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:46,464 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:46,464 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:49,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:49,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:49,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:52,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:52,513 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:52,513 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:55,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:55,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:55,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:53:58,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:53:58,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:53:58,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:01,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.02 % 2021-12-16 11:54:01,573 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:01,573 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:04,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.22 % 2021-12-16 11:54:04,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:04,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:07,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.22 % 2021-12-16 11:54:07,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:07,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:10,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:10,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:10,630 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:13,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:13,645 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:13,645 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:16,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:16,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:16,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:19,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:19,676 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:19,676 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:22,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:22,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:22,696 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:25,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:25,714 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:25,716 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:28,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:28,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:28,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:31,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:31,752 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:31,752 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:34,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:34,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:34,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:37,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:37,783 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:37,783 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:40,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:40,803 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:40,803 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:43,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:43,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:43,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:46,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:46,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:46,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:49,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:49,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:49,855 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:52,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:52,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:52,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:55,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:55,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:55,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:54:58,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.41 % 2021-12-16 11:54:58,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:54:58,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:01,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:01,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:01,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:04,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:04,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:04,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:07,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:07,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:07,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:10,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:10,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:10,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:13,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:13,994 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:13,994 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:17,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:17,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:17,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:20,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.61 % 2021-12-16 11:55:20,036 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:20,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:23,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.80 % 2021-12-16 11:55:23,053 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:23,053 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:26,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.80 % 2021-12-16 11:55:26,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:26,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:29,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.80 % 2021-12-16 11:55:29,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:29,096 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:32,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 15.80 % 2021-12-16 11:55:32,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:32,117 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:35,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:35,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:35,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:38,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:38,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:38,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:41,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:41,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:41,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:44,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:47,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:47,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:47,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:50,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:53,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:53,246 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:53,246 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:56,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:56,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:56,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:55:59,272 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:55:59,276 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:55:59,276 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:02,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:02,295 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:02,295 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:05,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:05,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:05,313 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:08,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:08,331 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:08,331 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:11,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:11,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:11,346 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:14,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:14,362 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:14,362 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:17,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:17,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:17,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:20,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:20,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:20,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:23,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:23,411 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:23,411 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:26,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:26,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:26,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:29,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:29,441 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:29,443 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:32,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:32,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:32,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:35,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.00 % 2021-12-16 11:56:35,506 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:35,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:38,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.19 % 2021-12-16 11:56:38,525 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:38,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:41,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.19 % 2021-12-16 11:56:41,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:41,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:44,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.19 % 2021-12-16 11:56:44,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:44,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:47,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.19 % 2021-12-16 11:56:47,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:47,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:50,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.39 % 2021-12-16 11:56:50,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:50,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:53,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.39 % 2021-12-16 11:56:53,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:53,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:56,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.39 % 2021-12-16 11:56:56,628 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:56,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:56:59,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.39 % 2021-12-16 11:56:59,644 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:56:59,644 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:02,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:02,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:02,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:05,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:05,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:05,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:08,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:08,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:08,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:11,732 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:11,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:11,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:14,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:14,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:14,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:17,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:17,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:17,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:20,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:20,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:20,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:23,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:23,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:23,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:26,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:26,849 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:26,849 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:29,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:29,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:29,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:32,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:32,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:32,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:35,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:35,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:35,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:38,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:38,918 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:38,920 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:41,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:41,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:41,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:44,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:44,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:44,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:47,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:47,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:47,971 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:50,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:50,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:50,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:53,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:54,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:54,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:57:57,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:57:57,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:57:57,030 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:00,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:58:00,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:00,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:03,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:58:03,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:03,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:06,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:58:06,079 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:06,079 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:09,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.58 % 2021-12-16 11:58:09,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:09,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:12,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.78 % 2021-12-16 11:58:12,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:12,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:15,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.97 % 2021-12-16 11:58:15,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:15,132 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:18,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 16.97 % 2021-12-16 11:58:18,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:18,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:21,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:21,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:21,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:24,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:24,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:24,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:27,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:27,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:27,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:30,220 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:30,232 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:30,232 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:33,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:33,253 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:33,253 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:36,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:36,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:36,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:39,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:39,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:39,285 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:42,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:42,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:42,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:45,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:45,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:45,329 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:48,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:48,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:48,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:51,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:51,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:51,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:54,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:54,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:54,394 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:58:57,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:58:57,410 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:58:57,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:00,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:00,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:00,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:03,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:03,446 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:03,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:06,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:06,466 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:06,466 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:09,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:09,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:09,484 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:12,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:12,499 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:12,499 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:15,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:15,515 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:15,515 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:18,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:18,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:18,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:21,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.17 % 2021-12-16 11:59:21,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:21,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:24,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 11:59:24,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:24,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:27,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 11:59:27,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:27,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:30,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 11:59:30,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:30,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:33,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.36 % 2021-12-16 11:59:33,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:33,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:36,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.56 % 2021-12-16 11:59:36,632 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:36,632 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:39,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.56 % 2021-12-16 11:59:39,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:39,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:42,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.56 % 2021-12-16 11:59:42,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:42,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:45,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 11:59:45,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:45,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:48,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 11:59:48,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:48,704 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:51,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 11:59:51,730 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:51,730 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:54,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 11:59:54,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:54,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 11:59:57,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 11:59:57,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 11:59:57,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:00,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:00,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:00,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:03,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:03,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:03,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:06,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:06,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:06,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:09,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:09,882 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:09,882 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:12,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:12,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:12,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:15,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:15,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:15,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:18,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:18,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:18,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:21,941 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:21,966 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:21,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:24,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:24,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:24,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:27,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:28,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:28,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:31,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:31,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:31,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:34,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:34,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:34,033 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:37,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:37,055 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:37,055 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:40,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.76 % 2021-12-16 12:00:40,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:40,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:43,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.95 % 2021-12-16 12:00:43,089 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:43,089 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:46,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.95 % 2021-12-16 12:00:46,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:46,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:49,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.95 % 2021-12-16 12:00:49,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:49,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:52,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 17.95 % 2021-12-16 12:00:52,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:52,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:55,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:00:55,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:55,158 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:00:58,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:00:58,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:00:58,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:01,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:01,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:01,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:04,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:04,211 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:04,211 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:07,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:07,227 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:07,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:10,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:10,246 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:10,246 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:13,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:13,267 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:16,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:16,290 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:16,290 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:19,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:19,306 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:19,306 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:22,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:22,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:22,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:25,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.15 % 2021-12-16 12:01:25,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:25,341 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:28,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:28,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:28,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:31,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:31,382 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:31,382 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:34,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:34,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:34,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:37,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:37,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:37,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:40,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:40,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:40,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:43,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:43,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:43,451 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:46,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:46,469 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:46,469 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:49,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:49,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:49,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:52,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:52,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:52,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:55,528 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:55,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:55,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:01:58,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:01:58,548 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:01:58,548 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:01,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.34 % 2021-12-16 12:02:01,572 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:01,572 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:04,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.53 % 2021-12-16 12:02:04,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:04,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:07,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.53 % 2021-12-16 12:02:07,614 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:07,614 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:10,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.53 % 2021-12-16 12:02:10,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:10,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:13,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.53 % 2021-12-16 12:02:13,645 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:13,647 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:16,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:16,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:16,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:19,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:19,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:19,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:22,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:22,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:22,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:25,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:25,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:25,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:28,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:28,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:28,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:31,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:31,755 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:31,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:34,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:34,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:34,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:37,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:37,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:37,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:40,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:40,841 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:40,841 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:43,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:43,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:43,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:46,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:46,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:46,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:49,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:49,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:49,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:52,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:52,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:52,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:55,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:55,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:55,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:02:58,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:02:58,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:02:58,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:01,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:03:01,994 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:01,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:05,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:03:05,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:05,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:08,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.73 % 2021-12-16 12:03:08,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:08,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:11,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.93 % 2021-12-16 12:03:11,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:11,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:14,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.93 % 2021-12-16 12:03:14,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:14,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:17,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.93 % 2021-12-16 12:03:17,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:17,091 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:20,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 18.93 % 2021-12-16 12:03:20,107 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:20,107 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:23,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:23,130 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:23,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:26,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:26,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:26,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:29,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:29,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:29,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:32,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:32,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:32,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:35,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:35,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:35,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:38,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:38,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:38,216 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:41,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:41,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:41,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:44,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:44,255 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:44,255 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:47,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.12 % 2021-12-16 12:03:47,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:47,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:50,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:03:50,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:50,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:53,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:03:53,303 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:53,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:56,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:03:56,322 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:56,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:03:59,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:03:59,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:03:59,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:02,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:02,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:02,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:05,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:05,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:05,377 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:08,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:08,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:08,392 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:11,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:11,415 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:11,415 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:14,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:14,434 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:14,434 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:17,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:17,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:17,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:20,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:20,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:20,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:23,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:23,494 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:23,494 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:26,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:26,512 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:26,513 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:29,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:29,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:29,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:32,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:32,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:32,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:35,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.32 % 2021-12-16 12:04:35,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:35,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:38,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.51 % 2021-12-16 12:04:38,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:38,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:41,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.51 % 2021-12-16 12:04:41,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:41,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:44,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.51 % 2021-12-16 12:04:44,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:44,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:47,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.51 % 2021-12-16 12:04:47,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:47,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:50,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.51 % 2021-12-16 12:04:50,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:50,663 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:53,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:04:53,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:53,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:56,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:04:56,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:56,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:04:59,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:04:59,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:04:59,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:02,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:05:02,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:02,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:05,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:05:05,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:05,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:08,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:05:08,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:08,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:11,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.71 % 2021-12-16 12:05:11,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:11,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:14,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:14,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:14,808 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:17,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:17,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:17,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:20,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:20,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:20,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:23,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:23,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:23,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:26,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:26,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:26,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:29,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:29,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:29,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:32,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:32,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:32,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:35,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:35,934 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:35,934 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:38,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:38,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:38,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:41,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:41,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:41,969 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:44,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:44,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:44,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:47,996 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:48,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:48,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:51,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:51,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:51,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:54,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:54,036 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:54,036 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:05:57,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 19.90 % 2021-12-16 12:05:57,055 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:05:57,055 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:00,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:00,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:00,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:03,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:03,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:03,085 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:06,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:06,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:06,102 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:09,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:09,117 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:09,117 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:12,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:12,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:12,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:15,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:15,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:15,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:18,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:18,170 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:18,170 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:21,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:21,186 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:21,186 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:24,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:24,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:24,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:27,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:27,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:27,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:30,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:30,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:30,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:33,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:33,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:33,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:36,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.10 % 2021-12-16 12:06:36,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:36,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:39,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:39,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:39,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:42,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:42,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:42,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:45,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:45,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:45,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:48,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:48,336 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:48,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:51,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:51,352 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:51,352 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:54,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:54,374 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:54,374 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:06:57,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:06:57,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:06:57,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:00,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:00,408 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:00,408 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:03,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:03,438 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:03,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:06,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:06,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:06,454 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:09,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:09,470 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:09,471 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:12,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:12,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:12,489 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:15,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:15,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:15,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:18,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:18,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:18,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:21,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.49 % 2021-12-16 12:07:21,540 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:21,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:24,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:24,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:24,556 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:27,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:27,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:27,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:30,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:30,591 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:30,591 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:33,599 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:33,607 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:33,607 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:36,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:36,628 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:36,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:39,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:39,642 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:39,642 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:42,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:42,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:42,658 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:45,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:45,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:45,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:48,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:48,698 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:48,698 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:51,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:51,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:51,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:54,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:54,737 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:54,737 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:07:57,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.68 % 2021-12-16 12:07:57,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:07:57,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:00,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-16 12:08:00,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:00,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:03,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-16 12:08:03,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:03,793 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:06,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-16 12:08:06,812 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:06,812 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:09,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 20.88 % 2021-12-16 12:08:09,829 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:09,829 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:12,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:12,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:12,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:15,857 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:15,864 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:15,864 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:18,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:18,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:18,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:21,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:21,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:21,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:24,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:24,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:24,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:27,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:27,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:27,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:30,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:30,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:30,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:33,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:33,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:33,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:36,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:36,994 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:36,994 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:40,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:40,015 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:40,015 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:43,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:43,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:43,029 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:46,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:46,045 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:46,045 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:49,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.07 % 2021-12-16 12:08:49,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:49,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:52,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 12:08:52,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:52,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:55,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 12:08:55,099 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:55,099 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:08:58,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 12:08:58,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:08:58,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:01,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.27 % 2021-12-16 12:09:01,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:01,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:04,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.46 % 2021-12-16 12:09:04,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:04,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:07,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.46 % 2021-12-16 12:09:07,165 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:07,165 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:10,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.66 % 2021-12-16 12:09:10,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:10,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:13,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.66 % 2021-12-16 12:09:13,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:13,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:16,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.66 % 2021-12-16 12:09:16,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:16,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:19,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.66 % 2021-12-16 12:09:19,244 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:19,244 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:22,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.66 % 2021-12-16 12:09:22,262 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:22,262 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:25,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.85 % 2021-12-16 12:09:25,276 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:25,276 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:28,286 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.85 % 2021-12-16 12:09:28,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:28,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:31,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 21.85 % 2021-12-16 12:09:31,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:31,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:34,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:34,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:34,329 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:37,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:37,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:37,368 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:40,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:40,385 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:40,387 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:43,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:43,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:43,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:46,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:46,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:46,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:49,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:49,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:49,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:52,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:52,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:52,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:55,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:55,471 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:55,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:09:58,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:09:58,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:09:58,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:01,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:01,507 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:01,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:04,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:04,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:04,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:07,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:07,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:07,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:10,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:10,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:10,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:13,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:13,573 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:13,573 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:16,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.05 % 2021-12-16 12:10:16,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:16,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:19,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.24 % 2021-12-16 12:10:19,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:19,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:22,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 12:10:22,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:22,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:25,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.44 % 2021-12-16 12:10:25,647 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:25,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:28,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.63 % 2021-12-16 12:10:28,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:28,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:31,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.63 % 2021-12-16 12:10:31,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:31,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:34,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.83 % 2021-12-16 12:10:34,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:34,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:37,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.83 % 2021-12-16 12:10:37,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:37,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:40,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.83 % 2021-12-16 12:10:40,737 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:40,737 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:43,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 22.83 % 2021-12-16 12:10:43,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:43,755 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:46,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.03 % 2021-12-16 12:10:46,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:46,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:49,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:10:49,795 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:49,795 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:52,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:10:52,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:52,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:55,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:10:55,831 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:55,831 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:10:58,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:10:58,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:10:58,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:01,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:01,867 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:01,867 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:04,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:04,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:04,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:07,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:07,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:07,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:10,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:10,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:10,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:13,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:13,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:13,928 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:16,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:16,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:16,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:19,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:19,964 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:19,966 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:22,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:22,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:22,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:25,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:25,999 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:25,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:29,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:29,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:29,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:32,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:32,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:32,035 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:35,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:35,051 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:35,051 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:38,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:38,066 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:38,066 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:41,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.22 % 2021-12-16 12:11:41,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:41,084 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:44,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.42 % 2021-12-16 12:11:44,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:44,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:47,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:11:47,112 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:47,114 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:50,122 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:11:50,132 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:50,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:53,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:11:53,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:53,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:56,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:11:56,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:56,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:11:59,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:11:59,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:11:59,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:02,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.61 % 2021-12-16 12:12:02,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:02,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:05,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 23.81 % 2021-12-16 12:12:05,221 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:05,221 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:08,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.00 % 2021-12-16 12:12:08,240 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:08,240 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:11,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.20 % 2021-12-16 12:12:11,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:11,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:14,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:14,293 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:14,293 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:17,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:17,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:17,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:20,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:20,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:20,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:23,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:23,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:23,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:26,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:26,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:26,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:29,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:29,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:29,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:32,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:32,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:32,392 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:35,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:35,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:35,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:38,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:38,424 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:38,424 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:41,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:41,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:41,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:44,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:44,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:44,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:47,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:47,477 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:47,477 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:50,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:50,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:50,494 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:53,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:53,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:53,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:56,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:56,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:56,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:12:59,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.39 % 2021-12-16 12:12:59,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:12:59,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:02,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 12:13:02,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:02,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:05,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 12:13:05,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:05,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:08,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 12:13:08,604 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:08,604 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:11,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.59 % 2021-12-16 12:13:11,619 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:11,619 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:14,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.78 % 2021-12-16 12:13:14,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:14,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:17,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.78 % 2021-12-16 12:13:17,658 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:17,658 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:20,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.98 % 2021-12-16 12:13:20,676 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:20,676 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:23,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 24.98 % 2021-12-16 12:13:23,690 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:23,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:26,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:26,704 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:26,704 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:29,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:29,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:29,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:32,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:32,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:32,757 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:35,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:35,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:35,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:38,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:38,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:38,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:41,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.17 % 2021-12-16 12:13:41,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:41,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:44,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.37 % 2021-12-16 12:13:44,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:44,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:47,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.37 % 2021-12-16 12:13:47,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:47,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:50,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:13:50,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:50,875 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:53,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:13:53,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:53,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:56,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:13:56,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:56,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:13:59,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:13:59,926 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:13:59,926 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:02,936 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:14:02,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:02,946 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:05,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:14:05,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:05,964 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:08,982 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:14:08,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:08,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:11,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:14:12,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:12,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:15,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.56 % 2021-12-16 12:14:15,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:15,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:18,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.76 % 2021-12-16 12:14:18,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:18,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:21,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.76 % 2021-12-16 12:14:21,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:21,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:24,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.76 % 2021-12-16 12:14:24,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:24,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:27,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.76 % 2021-12-16 12:14:27,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:27,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:30,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.76 % 2021-12-16 12:14:30,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:30,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:33,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.96 % 2021-12-16 12:14:33,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:33,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:36,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.96 % 2021-12-16 12:14:36,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:36,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:39,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 25.96 % 2021-12-16 12:14:39,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:39,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:42,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.15 % 2021-12-16 12:14:42,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:42,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:45,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.15 % 2021-12-16 12:14:45,193 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:45,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:48,204 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.15 % 2021-12-16 12:14:48,210 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:48,210 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:51,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:14:51,232 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:51,232 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:54,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:14:54,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:54,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:14:57,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:14:57,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:14:57,270 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:00,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:00,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:00,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:03,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:03,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:03,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:06,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:06,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:06,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:09,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:09,334 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:09,334 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:12,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:12,352 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:12,352 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:15,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.35 % 2021-12-16 12:15:15,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:15,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:18,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.54 % 2021-12-16 12:15:18,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:18,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:21,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.54 % 2021-12-16 12:15:21,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:21,400 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:24,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.54 % 2021-12-16 12:15:24,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:24,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:27,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.74 % 2021-12-16 12:15:27,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:27,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:30,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.74 % 2021-12-16 12:15:30,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:30,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:33,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.74 % 2021-12-16 12:15:33,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:33,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:36,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.74 % 2021-12-16 12:15:36,487 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:36,487 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:39,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-16 12:15:39,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:39,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:42,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-16 12:15:42,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:42,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:45,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 26.93 % 2021-12-16 12:15:45,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:45,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:48,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 12:15:48,563 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:48,563 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:51,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 12:15:51,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:51,579 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:54,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 12:15:54,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:54,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:15:57,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 12:15:57,614 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:15:57,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:00,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.13 % 2021-12-16 12:16:00,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:00,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:03,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:03,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:03,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:06,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:06,665 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:06,665 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:09,676 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:09,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:09,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:12,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:12,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:12,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:15,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:15,714 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:15,714 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:18,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:18,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:18,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:21,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:21,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:21,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:24,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:24,765 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:24,765 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:27,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:27,779 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:27,779 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:30,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.32 % 2021-12-16 12:16:30,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:30,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:33,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.52 % 2021-12-16 12:16:33,812 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:33,812 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:36,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.71 % 2021-12-16 12:16:36,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:36,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:39,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.71 % 2021-12-16 12:16:39,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:39,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:42,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.71 % 2021-12-16 12:16:42,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:42,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:45,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.71 % 2021-12-16 12:16:45,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:45,882 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:48,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:16:48,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:48,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:51,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:16:51,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:51,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:54,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:16:54,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:54,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:16:57,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:16:57,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:16:57,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:00,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:17:00,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:00,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:03,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:17:03,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:03,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:07,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 27.91 % 2021-12-16 12:17:07,009 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:07,009 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:10,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:10,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:10,026 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:13,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:13,045 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:13,045 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:16,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:16,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:16,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:19,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:19,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:19,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:22,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:22,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:22,096 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:25,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.10 % 2021-12-16 12:17:25,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:25,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:28,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.30 % 2021-12-16 12:17:28,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:28,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:31,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:31,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:31,160 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:34,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:34,176 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:34,176 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:37,186 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:37,191 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:37,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:40,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:40,211 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:40,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:43,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:43,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:43,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:46,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.49 % 2021-12-16 12:17:46,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:46,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:49,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.69 % 2021-12-16 12:17:49,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:49,265 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:52,275 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.69 % 2021-12-16 12:17:52,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:52,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:55,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.69 % 2021-12-16 12:17:55,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:55,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:17:58,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.88 % 2021-12-16 12:17:58,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:17:58,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:01,331 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.88 % 2021-12-16 12:18:01,339 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:01,339 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:04,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 28.88 % 2021-12-16 12:18:04,355 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:04,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:07,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.08 % 2021-12-16 12:18:07,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:07,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:10,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.08 % 2021-12-16 12:18:10,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:10,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:13,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.08 % 2021-12-16 12:18:13,403 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:13,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:16,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.08 % 2021-12-16 12:18:16,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:16,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:19,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.08 % 2021-12-16 12:18:19,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:19,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:22,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:22,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:22,476 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:25,487 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:25,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:25,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:28,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:28,512 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:28,512 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:31,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:31,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:31,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:34,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:34,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:34,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:37,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:37,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:37,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:40,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:40,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:40,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:43,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:43,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:43,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:46,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:46,614 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:46,614 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:49,624 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:49,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:49,632 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:52,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:52,653 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:52,653 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:55,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:55,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:55,668 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:18:58,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.28 % 2021-12-16 12:18:58,683 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:18:58,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:01,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.47 % 2021-12-16 12:19:01,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:01,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:04,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.47 % 2021-12-16 12:19:04,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:04,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:07,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.67 % 2021-12-16 12:19:07,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:07,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:10,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.67 % 2021-12-16 12:19:10,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:10,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:13,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.67 % 2021-12-16 12:19:13,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:13,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:16,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.67 % 2021-12-16 12:19:16,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:16,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:19,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 29.86 % 2021-12-16 12:19:19,801 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:19,801 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:22,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.06 % 2021-12-16 12:19:22,818 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:22,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:25,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.06 % 2021-12-16 12:19:25,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:25,855 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:28,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.06 % 2021-12-16 12:19:28,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:28,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:31,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.06 % 2021-12-16 12:19:31,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:31,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:34,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.06 % 2021-12-16 12:19:34,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:34,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:37,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.25 % 2021-12-16 12:19:37,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:37,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:40,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:40,950 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:40,950 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:43,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:43,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:43,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:46,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:46,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:46,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:50,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:50,006 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:50,006 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:53,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:53,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:53,022 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:56,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:56,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:56,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:19:59,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:19:59,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:19:59,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:02,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:02,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:02,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:05,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:05,086 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:05,086 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:08,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:08,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:08,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:11,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:11,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:11,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:14,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:14,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:14,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:17,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:17,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:17,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:20,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:20,170 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:20,170 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:23,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.45 % 2021-12-16 12:20:23,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:23,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:26,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 12:20:26,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:26,201 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:29,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 12:20:29,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:29,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:32,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 12:20:32,239 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:32,239 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:35,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.64 % 2021-12-16 12:20:35,255 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:35,255 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:38,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 30.84 % 2021-12-16 12:20:38,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:38,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:41,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.03 % 2021-12-16 12:20:41,289 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:41,289 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:44,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.03 % 2021-12-16 12:20:44,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:44,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:47,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.03 % 2021-12-16 12:20:47,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:47,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:50,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.23 % 2021-12-16 12:20:50,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:50,341 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:53,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-16 12:20:53,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:53,357 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:56,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-16 12:20:56,377 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:56,377 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:20:59,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-16 12:20:59,395 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:20:59,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:02,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-16 12:21:02,415 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:02,415 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:05,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.42 % 2021-12-16 12:21:05,430 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:05,430 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:08,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:08,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:08,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:11,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:11,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:11,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:14,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:14,482 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:14,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:17,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:17,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:17,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:20,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:20,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:20,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:23,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:23,535 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:23,535 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:26,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:26,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:26,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:29,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:29,568 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:29,568 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:32,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:32,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:32,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:35,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:35,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:35,599 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:38,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.62 % 2021-12-16 12:21:38,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:38,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:41,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:41,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:41,630 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:44,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:44,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:44,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:47,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:47,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:47,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:50,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:50,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:50,686 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:53,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:53,701 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:53,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:56,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:56,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:56,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:21:59,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:21:59,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:21:59,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:02,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:22:02,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:02,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:05,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 31.82 % 2021-12-16 12:22:05,772 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:05,772 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:08,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.01 % 2021-12-16 12:22:08,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:08,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:11,811 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.01 % 2021-12-16 12:22:11,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:11,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:14,826 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.01 % 2021-12-16 12:22:14,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:14,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:17,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.21 % 2021-12-16 12:22:17,849 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:17,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:20,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.21 % 2021-12-16 12:22:20,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:20,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:23,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.21 % 2021-12-16 12:22:23,882 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:23,882 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:26,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.40 % 2021-12-16 12:22:26,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:26,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:29,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.40 % 2021-12-16 12:22:29,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:29,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:32,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.60 % 2021-12-16 12:22:32,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:32,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:35,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:35,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:35,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:38,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:38,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:38,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:41,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:42,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:42,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:45,015 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:45,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:45,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:48,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:48,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:48,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:51,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:51,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:51,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:54,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:54,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:54,079 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:22:57,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.79 % 2021-12-16 12:22:57,096 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:22:57,096 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:00,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:00,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:00,112 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:03,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:03,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:03,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:06,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:06,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:06,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:09,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:09,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:09,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:12,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:12,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:12,181 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:15,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:15,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:15,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:18,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:18,211 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:18,211 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:21,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:21,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:21,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:24,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:24,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:24,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:27,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 32.99 % 2021-12-16 12:23:27,278 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:27,278 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:30,288 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.18 % 2021-12-16 12:23:30,293 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:30,293 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:33,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.18 % 2021-12-16 12:23:33,313 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:33,313 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:36,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.18 % 2021-12-16 12:23:36,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:36,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:39,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.38 % 2021-12-16 12:23:39,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:39,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:42,369 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.38 % 2021-12-16 12:23:42,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:42,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:45,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.38 % 2021-12-16 12:23:45,395 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:45,395 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:48,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.38 % 2021-12-16 12:23:48,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:48,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:51,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.38 % 2021-12-16 12:23:51,430 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:51,430 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:54,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.57 % 2021-12-16 12:23:54,448 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:54,448 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:23:57,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.57 % 2021-12-16 12:23:57,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:23:57,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:00,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.57 % 2021-12-16 12:24:00,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:00,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:03,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.57 % 2021-12-16 12:24:03,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:03,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:06,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.57 % 2021-12-16 12:24:06,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:06,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:09,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.77 % 2021-12-16 12:24:09,563 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:09,563 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:12,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.96 % 2021-12-16 12:24:12,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:12,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:15,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.96 % 2021-12-16 12:24:15,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:15,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:18,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.96 % 2021-12-16 12:24:18,617 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:18,617 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:21,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 33.96 % 2021-12-16 12:24:21,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:21,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:24,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:24,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:24,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:27,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:27,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:27,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:30,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:30,686 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:30,687 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:33,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:33,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:33,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:36,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:36,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:36,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:39,734 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.16 % 2021-12-16 12:24:39,740 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:39,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:42,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:42,755 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:42,755 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:45,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:45,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:45,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:48,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:48,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:48,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:51,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:51,811 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:51,811 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:54,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:54,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:54,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:24:57,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:24:57,844 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:24:57,844 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:00,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:00,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:00,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:03,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:03,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:03,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:06,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:06,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:06,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:09,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:09,920 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:09,920 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:12,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:12,934 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:12,934 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:15,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.36 % 2021-12-16 12:25:15,948 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:15,948 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:18,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.55 % 2021-12-16 12:25:18,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:18,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:21,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.55 % 2021-12-16 12:25:22,002 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:22,002 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:25,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.75 % 2021-12-16 12:25:25,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:25,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:28,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.75 % 2021-12-16 12:25:28,036 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:28,036 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:31,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.94 % 2021-12-16 12:25:31,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:31,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:34,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 34.94 % 2021-12-16 12:25:34,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:34,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:37,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.14 % 2021-12-16 12:25:37,084 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:37,084 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:40,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.14 % 2021-12-16 12:25:40,105 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:40,105 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:43,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.14 % 2021-12-16 12:25:43,119 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:43,119 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:46,130 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.14 % 2021-12-16 12:25:46,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:46,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:49,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.14 % 2021-12-16 12:25:49,150 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:49,150 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:52,158 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.33 % 2021-12-16 12:25:52,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:52,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:55,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.33 % 2021-12-16 12:25:55,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:55,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:25:58,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.33 % 2021-12-16 12:25:58,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:25:58,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:01,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.33 % 2021-12-16 12:26:01,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:01,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:04,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.33 % 2021-12-16 12:26:04,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:04,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:07,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:07,253 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:07,253 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:10,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:10,270 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:10,270 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:13,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:13,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:13,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:16,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:16,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:16,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:19,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:19,316 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:19,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:22,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:22,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:22,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:25,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:25,354 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:25,354 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:28,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:28,372 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:28,372 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:31,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:31,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:31,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:34,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:34,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:34,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:37,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:37,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:37,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:40,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.53 % 2021-12-16 12:26:40,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:40,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:43,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.72 % 2021-12-16 12:26:43,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:43,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:46,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.72 % 2021-12-16 12:26:46,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:46,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:49,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.72 % 2021-12-16 12:26:49,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:49,489 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:52,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.92 % 2021-12-16 12:26:52,505 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:52,505 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:55,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.92 % 2021-12-16 12:26:55,522 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:55,522 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:26:58,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.92 % 2021-12-16 12:26:58,540 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:26:58,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:01,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 35.92 % 2021-12-16 12:27:01,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:01,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:04,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:04,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:04,576 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:07,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:07,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:07,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:10,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:10,612 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:10,614 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:13,622 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:13,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:13,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:16,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:16,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:16,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:19,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.31 % 2021-12-16 12:27:19,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:19,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:22,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.50 % 2021-12-16 12:27:22,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:22,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:25,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.50 % 2021-12-16 12:27:25,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:25,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:28,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:28,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:28,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:31,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:31,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:31,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:34,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:34,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:34,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:37,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:37,775 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:37,775 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:40,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:40,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:40,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:43,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:43,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:43,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:46,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:46,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:46,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:49,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:49,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:49,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:52,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:52,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:52,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:55,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:55,878 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:55,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:27:58,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:27:58,892 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:27:58,892 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:01,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.70 % 2021-12-16 12:28:01,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:01,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:04,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 12:28:04,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:04,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:07,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 12:28:07,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:07,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:10,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 36.89 % 2021-12-16 12:28:10,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:10,977 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:13,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.09 % 2021-12-16 12:28:13,992 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:13,992 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:17,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.09 % 2021-12-16 12:28:17,012 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:17,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:20,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.09 % 2021-12-16 12:28:20,030 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:20,030 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:23,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.09 % 2021-12-16 12:28:23,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:23,048 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:26,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.09 % 2021-12-16 12:28:26,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:26,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:29,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.28 % 2021-12-16 12:28:29,076 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:29,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:32,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.48 % 2021-12-16 12:28:32,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:32,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:35,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.48 % 2021-12-16 12:28:35,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:35,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:38,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:38,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:38,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:41,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:41,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:41,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:44,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:44,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:44,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:47,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:47,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:47,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:50,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:50,216 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:50,216 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:53,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:53,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:53,234 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:56,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:56,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:56,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:28:59,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:28:59,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:28:59,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:02,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:29:02,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:02,285 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:05,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:29:05,298 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:05,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:08,308 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:29:08,316 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:08,316 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:11,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:29:11,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:11,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:14,346 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.68 % 2021-12-16 12:29:14,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:14,357 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:17,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.87 % 2021-12-16 12:29:17,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:17,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:20,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 37.87 % 2021-12-16 12:29:20,385 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:20,387 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:23,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.07 % 2021-12-16 12:29:23,400 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:23,400 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:26,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:26,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:26,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:29,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:29,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:29,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:32,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:32,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:32,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:35,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:35,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:35,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:38,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:38,482 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:38,482 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:41,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:41,502 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:41,502 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:44,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:44,520 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:44,522 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:47,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:47,543 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:47,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:50,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.26 % 2021-12-16 12:29:50,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:50,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:53,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.65 % 2021-12-16 12:29:53,576 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:53,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:56,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.65 % 2021-12-16 12:29:56,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:56,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:29:59,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.65 % 2021-12-16 12:29:59,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:29:59,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:02,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:02,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:02,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:05,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:05,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:05,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:08,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:08,678 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:08,678 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:11,688 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:11,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:11,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:14,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:14,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:14,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:17,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:17,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:17,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:20,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:20,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:20,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:23,762 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:23,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:23,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:26,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:26,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:26,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:29,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 38.85 % 2021-12-16 12:30:29,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:29,801 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:32,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-16 12:30:32,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:32,816 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:35,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.04 % 2021-12-16 12:30:35,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:35,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:38,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:38,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:38,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:41,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:41,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:41,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:44,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:44,885 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:44,885 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:47,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:47,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:47,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:50,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:50,917 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:50,917 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:53,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:53,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:53,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:56,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:56,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:56,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:30:59,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:30:59,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:30:59,969 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:02,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:31:02,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:02,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:05,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:31:06,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:06,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:09,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:31:09,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:09,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:12,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.24 % 2021-12-16 12:31:12,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:12,033 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:15,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.63 % 2021-12-16 12:31:15,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:15,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:18,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.63 % 2021-12-16 12:31:18,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:18,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:21,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.82 % 2021-12-16 12:31:21,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:21,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:24,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.82 % 2021-12-16 12:31:24,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:24,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:27,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 39.82 % 2021-12-16 12:31:27,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:27,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:30,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:30,135 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:30,137 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:33,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:33,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:33,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:36,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:36,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:36,168 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:39,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:39,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:39,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:42,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:42,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:42,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:45,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:45,224 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:45,224 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:48,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:48,239 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:48,239 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:51,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:51,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:51,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:54,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:54,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:54,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:31:57,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.02 % 2021-12-16 12:31:57,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:31:57,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:00,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:00,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:00,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:03,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:03,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:03,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:06,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:06,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:06,341 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:09,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:09,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:09,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:12,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:12,374 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:12,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:15,385 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:15,395 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:15,395 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:18,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:18,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:18,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:21,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:21,430 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:21,431 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:24,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:24,446 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:24,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:27,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:27,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:27,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:30,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:30,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:30,489 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:33,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:33,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:33,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:36,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.41 % 2021-12-16 12:32:36,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:36,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:39,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:39,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:39,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:42,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:42,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:42,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:45,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:45,584 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:45,584 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:48,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:48,602 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:48,602 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:51,614 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:51,624 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:51,625 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:54,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.61 % 2021-12-16 12:32:54,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:54,640 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:32:57,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.80 % 2021-12-16 12:32:57,658 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:32:57,658 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:00,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 40.80 % 2021-12-16 12:33:00,677 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:00,677 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:03,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.00 % 2021-12-16 12:33:03,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:03,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:06,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 12:33:06,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:06,737 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:09,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 12:33:09,757 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:09,757 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:12,767 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 12:33:12,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:12,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:15,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.39 % 2021-12-16 12:33:15,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:15,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:18,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:18,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:18,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:21,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:21,821 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:21,821 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:24,829 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:24,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:24,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:27,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:27,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:27,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:30,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:30,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:30,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:33,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:33,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:33,895 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:36,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:36,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:36,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:39,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:39,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:39,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:42,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:42,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:42,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:45,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.58 % 2021-12-16 12:33:45,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:45,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:48,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:33:49,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:49,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:52,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:33:52,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:52,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:55,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:33:55,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:55,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:33:58,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:33:58,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:33:58,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:01,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:01,081 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:01,081 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:04,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:04,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:04,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:07,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:07,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:07,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:10,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:10,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:10,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:13,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:13,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:13,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:16,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:16,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:16,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:19,176 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:19,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:19,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:22,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:22,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:22,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:25,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:25,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:25,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:28,219 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.78 % 2021-12-16 12:34:28,226 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:28,226 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:31,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 41.97 % 2021-12-16 12:34:31,243 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:31,243 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:34,266 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.17 % 2021-12-16 12:34:34,272 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:34,272 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:37,280 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.36 % 2021-12-16 12:34:37,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:37,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:40,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.56 % 2021-12-16 12:34:40,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:40,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:43,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:43,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:43,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:46,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:46,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:46,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:49,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:49,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:49,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:52,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:52,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:52,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:55,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:55,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:55,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:34:58,428 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.75 % 2021-12-16 12:34:58,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:34:58,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:01,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:01,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:01,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:04,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:04,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:04,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:07,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:07,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:07,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:10,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:10,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:10,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:13,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:13,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:13,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:16,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:16,545 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:16,545 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:19,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:19,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:19,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:22,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:22,584 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:22,584 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:25,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:25,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:25,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:28,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:28,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:28,622 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:31,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:31,645 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:31,645 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:34,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:34,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:34,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:37,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:37,687 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:37,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:40,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:40,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:40,706 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:43,714 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:43,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:43,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:46,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 42.95 % 2021-12-16 12:35:46,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:46,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:49,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.15 % 2021-12-16 12:35:49,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:49,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:52,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.15 % 2021-12-16 12:35:52,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:52,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:55,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.15 % 2021-12-16 12:35:55,803 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:55,803 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:35:58,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.15 % 2021-12-16 12:35:58,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:35:58,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:01,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.15 % 2021-12-16 12:36:01,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:01,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:04,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.34 % 2021-12-16 12:36:04,854 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:04,854 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:07,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.34 % 2021-12-16 12:36:07,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:07,875 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:10,884 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.93 % 2021-12-16 12:36:10,890 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:10,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:13,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 43.93 % 2021-12-16 12:36:13,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:13,908 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:16,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:16,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:16,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:19,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:19,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:19,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:22,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:22,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:22,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:25,992 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:26,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:26,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:29,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:29,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:29,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:32,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:32,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:32,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:35,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:35,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:35,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:38,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:38,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:38,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:41,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:41,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:41,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:44,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:44,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:44,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:47,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:47,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:47,132 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:50,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.12 % 2021-12-16 12:36:50,150 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:50,150 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:53,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:36:53,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:53,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:56,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:36:56,186 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:56,186 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:36:59,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:36:59,206 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:36:59,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:02,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:02,221 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:02,221 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:05,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:05,239 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:05,239 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:08,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:08,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:08,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:11,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:11,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:11,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:14,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:14,311 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:14,311 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:17,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:17,331 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:17,331 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:20,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:20,346 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:20,346 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:23,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:23,362 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:23,364 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:26,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.32 % 2021-12-16 12:37:26,382 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:26,382 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:29,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.51 % 2021-12-16 12:37:29,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:29,405 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:32,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.51 % 2021-12-16 12:37:32,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:32,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:35,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.51 % 2021-12-16 12:37:35,446 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:35,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:38,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.71 % 2021-12-16 12:37:38,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:38,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:41,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 44.90 % 2021-12-16 12:37:41,482 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:41,482 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:44,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.10 % 2021-12-16 12:37:44,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:44,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:47,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.10 % 2021-12-16 12:37:47,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:47,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:50,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:37:50,545 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:50,545 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:53,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:37:53,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:53,563 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:56,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:37:56,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:56,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:37:59,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:37:59,604 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:37:59,604 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:02,617 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:38:02,624 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:02,624 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:05,635 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:38:05,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:05,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:08,653 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:38:08,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:08,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:11,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:38:11,678 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:11,678 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:14,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.29 % 2021-12-16 12:38:14,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:14,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:17,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:17,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:17,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:20,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:20,739 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:20,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:23,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:23,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:23,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:26,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:26,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:26,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:29,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:29,798 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:29,798 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:32,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:32,841 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:32,841 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:35,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:35,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:35,858 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:38,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:38,874 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:38,874 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:41,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:41,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:41,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:44,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:44,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:44,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:47,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:47,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:47,941 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:50,954 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:50,964 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:50,964 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:53,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:53,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:53,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:38:56,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.49 % 2021-12-16 12:38:56,999 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:38:56,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:00,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.68 % 2021-12-16 12:39:00,019 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:00,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:03,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.68 % 2021-12-16 12:39:03,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:03,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:06,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 45.88 % 2021-12-16 12:39:06,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:06,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:09,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.08 % 2021-12-16 12:39:09,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:09,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:12,104 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.08 % 2021-12-16 12:39:12,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:12,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:15,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.08 % 2021-12-16 12:39:15,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:15,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:18,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.27 % 2021-12-16 12:39:18,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:18,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:21,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 12:39:21,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:21,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:24,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 12:39:24,191 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:24,191 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:27,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 12:39:27,210 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:27,210 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:30,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 12:39:30,229 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:30,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:33,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.47 % 2021-12-16 12:39:33,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:33,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:36,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:36,270 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:36,270 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:39,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:39,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:39,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:42,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:42,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:42,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:45,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:45,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:45,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:48,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:48,344 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:48,344 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:51,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:51,359 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:51,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:54,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:54,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:54,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:39:57,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:39:57,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:39:57,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:00,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:40:00,424 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:00,424 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:03,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:40:03,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:03,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:06,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:40:06,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:06,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:09,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:40:09,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:09,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:12,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.66 % 2021-12-16 12:40:12,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:12,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:15,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.86 % 2021-12-16 12:40:15,520 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:15,520 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:18,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.86 % 2021-12-16 12:40:18,538 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:18,538 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:21,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.86 % 2021-12-16 12:40:21,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:21,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:24,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.86 % 2021-12-16 12:40:24,594 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:24,594 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:27,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 46.86 % 2021-12-16 12:40:27,619 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:27,619 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:30,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:30,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:30,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:33,650 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:33,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:33,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:36,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:36,670 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:36,670 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:39,686 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:39,691 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:39,691 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:42,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:42,729 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:42,729 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:45,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:45,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:45,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:48,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.05 % 2021-12-16 12:40:48,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:48,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:51,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.44 % 2021-12-16 12:40:51,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:51,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:54,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.44 % 2021-12-16 12:40:54,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:54,798 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:40:57,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:40:57,816 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:40:57,818 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:00,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:00,839 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:00,839 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:03,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:03,859 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:03,859 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:06,869 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:06,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:06,875 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:09,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:09,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:09,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:12,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:12,910 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:12,911 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:15,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:15,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:15,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:18,938 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:18,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:18,946 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:21,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:21,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:21,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:24,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 47.83 % 2021-12-16 12:41:24,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:24,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:27,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:28,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:28,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:31,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:31,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:31,022 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:34,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:34,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:34,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:37,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:37,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:37,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:40,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:40,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:40,073 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:43,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:43,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:43,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:46,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:46,105 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:46,105 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:49,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:49,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:49,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:52,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:52,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:52,147 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:55,157 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:55,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:55,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:41:58,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:41:58,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:41:58,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:01,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.03 % 2021-12-16 12:42:01,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:01,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:04,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.23 % 2021-12-16 12:42:04,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:04,216 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:07,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 12:42:07,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:07,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:10,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 12:42:10,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:10,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:13,267 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 12:42:13,273 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:13,273 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:16,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.42 % 2021-12-16 12:42:16,290 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:16,290 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:19,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.62 % 2021-12-16 12:42:19,306 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:19,306 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:22,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.62 % 2021-12-16 12:42:22,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:22,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:25,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.81 % 2021-12-16 12:42:25,354 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:25,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:28,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.81 % 2021-12-16 12:42:28,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:28,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:31,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 48.81 % 2021-12-16 12:42:31,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:31,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:34,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:34,415 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:34,415 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:37,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:37,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:37,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:40,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:40,464 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:40,464 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:43,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:43,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:43,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:46,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:46,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:46,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:49,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:49,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:49,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:52,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.01 % 2021-12-16 12:42:52,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:52,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:55,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:42:55,563 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:55,563 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:42:58,573 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:42:58,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:42:58,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:01,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:43:01,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:01,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:04,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:43:04,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:04,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:07,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:43:07,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:07,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:10,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.20 % 2021-12-16 12:43:10,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:10,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:13,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:13,668 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:13,668 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:16,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:16,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:16,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:19,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:19,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:19,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:22,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:22,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:22,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:25,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:25,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:25,757 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:28,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:28,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:28,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:31,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:31,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:31,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:34,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:34,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:34,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:37,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:37,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:37,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:40,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:40,852 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:40,852 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:43,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:43,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:43,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:46,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.40 % 2021-12-16 12:43:46,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:46,885 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:49,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.79 % 2021-12-16 12:43:49,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:49,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:52,915 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.79 % 2021-12-16 12:43:52,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:52,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:55,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.79 % 2021-12-16 12:43:55,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:55,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:43:58,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-16 12:43:58,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:43:58,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:01,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 49.98 % 2021-12-16 12:44:01,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:01,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:04,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.18 % 2021-12-16 12:44:04,984 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:04,984 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:07,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.18 % 2021-12-16 12:44:08,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:08,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:11,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.18 % 2021-12-16 12:44:11,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:11,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:14,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.18 % 2021-12-16 12:44:14,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:14,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:17,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.37 % 2021-12-16 12:44:17,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:17,053 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:20,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.37 % 2021-12-16 12:44:20,068 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:20,068 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:23,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.37 % 2021-12-16 12:44:23,084 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:23,084 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:26,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.37 % 2021-12-16 12:44:26,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:26,102 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:29,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.37 % 2021-12-16 12:44:29,122 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:29,122 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:32,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:32,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:32,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:35,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:35,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:35,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:38,165 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:38,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:38,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:41,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:41,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:41,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:44,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:44,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:44,214 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:47,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:47,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:47,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:50,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:50,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:50,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:53,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:53,276 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:53,276 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:56,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:56,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:56,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:44:59,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:44:59,321 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:44:59,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:02,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:02,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:02,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:05,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:05,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:05,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:08,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:08,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:08,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:11,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:11,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:11,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:14,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:14,424 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:14,424 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:17,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:17,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:17,441 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:20,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:20,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:20,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:23,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.57 % 2021-12-16 12:45:23,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:23,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:26,482 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.96 % 2021-12-16 12:45:26,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:26,489 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:29,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 50.96 % 2021-12-16 12:45:29,512 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:29,512 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:32,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.15 % 2021-12-16 12:45:32,535 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:32,536 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:35,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.15 % 2021-12-16 12:45:35,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:35,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:38,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.35 % 2021-12-16 12:45:38,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:38,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:41,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.54 % 2021-12-16 12:45:41,581 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:41,581 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:44,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:44,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:44,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:47,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:47,619 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:47,619 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:50,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:50,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:50,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:53,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:53,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:53,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:56,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:56,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:56,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:45:59,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:45:59,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:45:59,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:02,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:02,714 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:02,714 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:05,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:05,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:05,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:08,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:08,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:08,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:11,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:11,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:11,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:14,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:14,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:14,789 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:17,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:17,808 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:17,808 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:20,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:20,826 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:20,826 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:23,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:23,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:23,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:26,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:26,867 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:26,867 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:29,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:29,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:29,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:32,903 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:32,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:32,910 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:35,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:35,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:35,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:38,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:38,944 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:38,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:41,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:41,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:41,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:44,971 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:44,977 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:44,977 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:47,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.74 % 2021-12-16 12:46:47,992 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:47,992 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:51,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.94 % 2021-12-16 12:46:51,010 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:51,010 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:54,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 51.94 % 2021-12-16 12:46:54,030 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:54,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:46:57,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.13 % 2021-12-16 12:46:57,055 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:46:57,055 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:00,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.13 % 2021-12-16 12:47:00,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:00,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:03,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.33 % 2021-12-16 12:47:03,089 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:03,089 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:06,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.33 % 2021-12-16 12:47:06,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:06,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:09,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.72 % 2021-12-16 12:47:09,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:09,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:12,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.72 % 2021-12-16 12:47:12,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:12,142 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:15,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.72 % 2021-12-16 12:47:15,157 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:15,157 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:18,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:18,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:18,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:21,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:21,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:21,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:24,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:24,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:24,204 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:27,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:27,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:27,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:30,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:30,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:30,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:33,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:33,256 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:33,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:36,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:36,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:36,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:39,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:39,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:39,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:42,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:42,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:42,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:45,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:45,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:45,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:48,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:48,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:48,338 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:51,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:51,352 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:51,352 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:54,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:54,377 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:54,377 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:47:57,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:47:57,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:47:57,394 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:00,404 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:48:00,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:00,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:03,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:48:03,431 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:03,433 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:06,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 52.91 % 2021-12-16 12:48:06,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:06,447 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:09,457 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.11 % 2021-12-16 12:48:09,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:09,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:12,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.11 % 2021-12-16 12:48:12,490 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:12,490 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:15,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.11 % 2021-12-16 12:48:15,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:15,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:18,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.11 % 2021-12-16 12:48:18,538 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:18,539 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:21,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.11 % 2021-12-16 12:48:21,553 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:21,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:24,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.30 % 2021-12-16 12:48:24,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:24,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:27,579 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.50 % 2021-12-16 12:48:27,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:27,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:30,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.50 % 2021-12-16 12:48:30,605 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:30,607 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:33,615 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.50 % 2021-12-16 12:48:33,624 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:33,624 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:36,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:36,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:36,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:39,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:39,665 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:39,665 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:42,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:42,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:42,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:45,690 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:45,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:45,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:48,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:48,721 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:48,721 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:51,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.69 % 2021-12-16 12:48:51,739 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:51,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:54,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.89 % 2021-12-16 12:48:54,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:54,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:48:57,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.89 % 2021-12-16 12:48:57,770 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:48:57,770 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:00,780 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 53.89 % 2021-12-16 12:49:00,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:00,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:03,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:03,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:03,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:06,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:06,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:06,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:09,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:09,841 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:09,841 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:12,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:12,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:12,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:15,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:15,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:15,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:18,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.08 % 2021-12-16 12:49:18,890 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:18,890 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:21,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:21,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:21,908 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:24,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:24,928 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:24,928 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:27,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:27,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:27,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:30,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:30,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:30,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:33,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:33,977 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:33,977 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:36,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:36,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:36,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:40,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:40,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:40,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:43,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:43,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:43,029 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:46,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:46,043 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:46,043 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:49,053 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:49,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:49,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:52,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:52,076 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:52,076 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:55,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.28 % 2021-12-16 12:49:55,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:55,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:49:58,107 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:49:58,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:49:58,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:01,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:01,127 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:01,127 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:04,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:04,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:04,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:07,163 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:07,171 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:07,171 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:10,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:10,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:10,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:13,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:13,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:13,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:16,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:16,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:16,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:19,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:19,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:19,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:22,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:22,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:22,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:25,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 54.87 % 2021-12-16 12:50:25,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:25,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:28,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.06 % 2021-12-16 12:50:28,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:28,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:31,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.06 % 2021-12-16 12:50:31,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:31,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:34,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.26 % 2021-12-16 12:50:34,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:34,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:37,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:37,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:37,338 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:40,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:40,354 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:40,354 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:43,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:43,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:43,368 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:46,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:46,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:46,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:49,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:49,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:49,403 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:52,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:52,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:52,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:55,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:55,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:55,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:50:58,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:50:58,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:50:58,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:01,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:51:01,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:01,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:04,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:51:04,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:04,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:07,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:51:07,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:07,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:10,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:51:10,522 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:10,522 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:13,530 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.45 % 2021-12-16 12:51:13,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:13,536 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:16,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.65 % 2021-12-16 12:51:16,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:16,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:19,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.84 % 2021-12-16 12:51:19,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:19,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:22,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.84 % 2021-12-16 12:51:22,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:22,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:25,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.84 % 2021-12-16 12:51:25,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:25,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:28,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.84 % 2021-12-16 12:51:28,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:28,617 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:31,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 55.84 % 2021-12-16 12:51:31,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:31,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:34,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.04 % 2021-12-16 12:51:34,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:34,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:37,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.04 % 2021-12-16 12:51:37,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:37,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:40,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.04 % 2021-12-16 12:51:40,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:40,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:43,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.04 % 2021-12-16 12:51:43,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:43,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:46,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.04 % 2021-12-16 12:51:46,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:46,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:49,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 12:51:49,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:49,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:52,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 12:51:52,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:52,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:55,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 12:51:55,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:55,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:51:58,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 12:51:58,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:51:58,808 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:01,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.23 % 2021-12-16 12:52:01,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:01,823 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:04,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.43 % 2021-12-16 12:52:04,839 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:04,839 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:07,849 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.43 % 2021-12-16 12:52:07,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:07,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:10,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:10,878 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:10,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:13,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:13,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:13,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:16,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:16,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:16,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:19,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:19,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:19,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:22,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:22,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:22,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:25,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:25,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:25,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:28,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.62 % 2021-12-16 12:52:28,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:28,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:31,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:31,999 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:31,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:35,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:35,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:35,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:38,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:38,028 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:38,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:41,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:41,045 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:41,046 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:44,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:44,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:44,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:47,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:47,086 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:47,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:50,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:50,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:50,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:53,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:53,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:53,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:56,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 56.82 % 2021-12-16 12:52:56,135 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:56,135 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:52:59,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:52:59,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:52:59,160 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:02,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:53:02,188 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:02,190 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:05,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:53:05,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:05,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:08,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:53:08,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:08,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:11,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:53:11,234 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:11,234 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:14,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.21 % 2021-12-16 12:53:14,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:14,253 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:17,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:17,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:17,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:20,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:20,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:20,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:23,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:23,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:23,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:26,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:26,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:26,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:29,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:29,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:29,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:32,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.41 % 2021-12-16 12:53:32,352 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:32,352 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:35,364 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.60 % 2021-12-16 12:53:35,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:35,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:38,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.60 % 2021-12-16 12:53:38,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:38,384 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:41,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.60 % 2021-12-16 12:53:41,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:41,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:44,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.60 % 2021-12-16 12:53:44,441 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:44,441 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:47,451 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.80 % 2021-12-16 12:53:47,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:47,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:50,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:53:50,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:50,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:53,492 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:53:53,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:53,497 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:56,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:53:56,512 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:56,512 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:53:59,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:53:59,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:53:59,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:02,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:02,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:02,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:05,581 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:05,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:05,589 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:08,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:08,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:08,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:11,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:11,622 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:11,624 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:14,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:14,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:14,640 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:17,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:17,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:17,655 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:20,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 57.99 % 2021-12-16 12:54:20,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:20,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:23,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:23,693 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:23,693 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:26,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:26,709 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:26,709 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:29,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:29,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:29,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:32,733 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:32,742 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:32,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:35,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:35,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:35,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:38,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.19 % 2021-12-16 12:54:38,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:38,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:41,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.38 % 2021-12-16 12:54:41,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:41,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:44,813 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.58 % 2021-12-16 12:54:44,821 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:44,821 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:47,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.58 % 2021-12-16 12:54:47,841 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:47,841 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:50,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.58 % 2021-12-16 12:54:50,858 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:50,858 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:53,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.58 % 2021-12-16 12:54:53,885 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:53,887 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:56,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.58 % 2021-12-16 12:54:56,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:56,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:54:59,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.77 % 2021-12-16 12:54:59,918 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:54:59,918 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:02,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.77 % 2021-12-16 12:55:02,934 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:02,934 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:05,944 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.97 % 2021-12-16 12:55:05,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:05,954 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:08,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.97 % 2021-12-16 12:55:08,971 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:08,971 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:11,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.97 % 2021-12-16 12:55:11,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:11,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:15,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.97 % 2021-12-16 12:55:15,007 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:15,007 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:18,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 58.97 % 2021-12-16 12:55:18,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:18,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:21,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:21,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:21,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:24,048 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:24,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:24,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:27,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:27,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:27,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:30,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:30,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:30,094 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:33,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:33,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:33,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:36,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:36,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:36,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:39,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:39,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:39,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:42,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:42,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:42,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:45,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:45,181 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:45,181 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:48,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:48,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:48,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:51,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:51,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:51,216 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:54,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:54,232 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:54,232 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:55:57,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.16 % 2021-12-16 12:55:57,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:55:57,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:00,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.36 % 2021-12-16 12:56:00,270 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:00,272 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:03,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.36 % 2021-12-16 12:56:03,295 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:03,295 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:06,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.55 % 2021-12-16 12:56:06,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:06,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:09,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.75 % 2021-12-16 12:56:09,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:09,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:12,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:12,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:12,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:15,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:15,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:15,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:18,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:18,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:18,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:21,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:21,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:21,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:24,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:24,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:24,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:27,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:27,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:27,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:30,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:30,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:30,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:33,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:33,479 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:33,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:36,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:36,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:36,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:39,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 59.95 % 2021-12-16 12:56:39,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:39,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:42,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.14 % 2021-12-16 12:56:42,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:42,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:45,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.14 % 2021-12-16 12:56:45,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:45,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:48,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.14 % 2021-12-16 12:56:48,569 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:48,569 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:51,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.14 % 2021-12-16 12:56:51,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:51,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:54,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.14 % 2021-12-16 12:56:54,612 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:54,614 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:56:57,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:56:57,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:56:57,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:00,642 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:00,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:00,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:03,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:03,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:03,668 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:06,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:06,687 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:06,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:09,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:09,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:09,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:12,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:12,723 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:12,723 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:15,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:15,740 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:15,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:18,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.34 % 2021-12-16 12:57:18,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:18,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:21,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.53 % 2021-12-16 12:57:21,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:21,778 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:24,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.73 % 2021-12-16 12:57:24,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:24,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:27,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.73 % 2021-12-16 12:57:27,821 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:27,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:30,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:30,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:30,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:33,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:33,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:33,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:36,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:36,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:36,875 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:39,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:39,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:39,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:42,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:42,917 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:42,917 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:45,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 60.92 % 2021-12-16 12:57:45,936 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:45,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:48,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:57:48,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:48,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:51,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:57:51,974 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:51,974 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:54,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:57:54,994 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:54,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:57:58,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:57:58,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:57:58,026 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:01,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:01,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:01,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:04,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:04,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:04,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:07,066 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:07,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:07,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:10,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:10,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:10,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:13,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:13,109 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:13,109 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:16,119 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.12 % 2021-12-16 12:58:16,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:16,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:19,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.31 % 2021-12-16 12:58:19,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:19,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:22,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.31 % 2021-12-16 12:58:22,160 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:22,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:25,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.31 % 2021-12-16 12:58:25,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:25,176 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:28,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.31 % 2021-12-16 12:58:28,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:28,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:31,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.51 % 2021-12-16 12:58:31,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:31,216 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:34,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.51 % 2021-12-16 12:58:34,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:34,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:37,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.70 % 2021-12-16 12:58:37,244 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:37,244 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:40,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.70 % 2021-12-16 12:58:40,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:40,265 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:43,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.90 % 2021-12-16 12:58:43,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:43,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:46,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 61.90 % 2021-12-16 12:58:46,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:46,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:49,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:58:49,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:49,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:52,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:58:52,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:52,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:55,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:58:55,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:55,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:58:58,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:58:58,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:58:58,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:01,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:01,400 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:01,400 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:04,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:04,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:04,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:07,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:07,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:07,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:10,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:10,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:10,454 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:13,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:13,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:13,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:16,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:16,490 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:16,490 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:19,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.09 % 2021-12-16 12:59:19,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:19,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:22,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:22,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:22,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:25,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:25,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:25,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:28,555 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:28,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:28,561 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:31,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:31,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:31,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:34,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:34,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:34,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:37,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:37,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:37,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:40,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.29 % 2021-12-16 12:59:40,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:40,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:43,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.48 % 2021-12-16 12:59:43,645 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:43,645 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:46,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.48 % 2021-12-16 12:59:46,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:46,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:49,670 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.48 % 2021-12-16 12:59:49,678 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:49,678 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:52,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.68 % 2021-12-16 12:59:52,696 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:52,696 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:55,706 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.68 % 2021-12-16 12:59:55,711 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:55,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 12:59:58,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.68 % 2021-12-16 12:59:58,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 12:59:58,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:01,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.68 % 2021-12-16 13:00:01,743 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:01,743 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:04,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.87 % 2021-12-16 13:00:04,760 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:04,762 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:07,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.87 % 2021-12-16 13:00:07,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:07,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:10,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 62.87 % 2021-12-16 13:00:10,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:10,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:13,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.07 % 2021-12-16 13:00:13,818 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:13,818 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:16,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.07 % 2021-12-16 13:00:16,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:16,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:19,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.07 % 2021-12-16 13:00:19,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:19,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:22,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.07 % 2021-12-16 13:00:22,890 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:22,890 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:25,901 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:25,908 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:25,908 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:28,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:28,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:28,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:31,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:31,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:31,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:34,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:34,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:34,954 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:37,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:37,974 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:37,974 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:40,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:40,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:40,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:44,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:44,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:44,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:47,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:47,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:47,030 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:50,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.27 % 2021-12-16 13:00:50,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:50,048 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:53,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.46 % 2021-12-16 13:00:53,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:53,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:56,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.46 % 2021-12-16 13:00:56,079 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:56,079 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:00:59,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.46 % 2021-12-16 13:00:59,099 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:00:59,099 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:02,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.66 % 2021-12-16 13:01:02,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:02,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:05,124 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.66 % 2021-12-16 13:01:05,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:05,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:08,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.66 % 2021-12-16 13:01:08,147 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:08,147 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:11,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.66 % 2021-12-16 13:01:11,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:11,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:14,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:14,180 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:14,180 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:17,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:17,198 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:17,198 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:20,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:20,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:20,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:23,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:23,229 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:23,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:26,237 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:26,247 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:26,247 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:29,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 63.85 % 2021-12-16 13:01:29,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:29,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:32,283 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.05 % 2021-12-16 13:01:32,293 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:32,295 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:35,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:35,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:35,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:38,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:38,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:38,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:41,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:41,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:41,338 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:44,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:44,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:44,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:47,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:47,377 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:47,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:50,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:50,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:50,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:53,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:53,405 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:53,405 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:56,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:56,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:56,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:01:59,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:01:59,438 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:01:59,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:02,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:02:02,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:02,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:05,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:02:05,479 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:05,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:08,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:02:08,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:08,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:11,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.24 % 2021-12-16 13:02:11,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:11,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:14,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.44 % 2021-12-16 13:02:14,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:14,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:17,538 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.63 % 2021-12-16 13:02:17,543 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:17,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:20,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.63 % 2021-12-16 13:02:20,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:20,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:23,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.63 % 2021-12-16 13:02:23,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:23,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:26,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.63 % 2021-12-16 13:02:26,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:26,589 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:29,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.63 % 2021-12-16 13:02:29,602 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:29,604 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:32,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.83 % 2021-12-16 13:02:32,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:32,619 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:35,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 64.83 % 2021-12-16 13:02:35,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:35,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:38,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.02 % 2021-12-16 13:02:38,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:38,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:41,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.02 % 2021-12-16 13:02:41,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:41,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:44,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.02 % 2021-12-16 13:02:44,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:44,703 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:47,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.02 % 2021-12-16 13:02:47,729 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:47,729 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:50,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.02 % 2021-12-16 13:02:50,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:50,755 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:53,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:02:53,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:53,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:56,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:02:56,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:56,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:02:59,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:02:59,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:02:59,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:02,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:02,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:02,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:05,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:05,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:05,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:08,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:08,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:08,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:11,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:11,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:11,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:14,897 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:14,903 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:14,903 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:17,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.22 % 2021-12-16 13:03:17,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:17,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:20,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:20,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:20,946 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:23,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:23,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:23,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:26,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:26,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:26,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:29,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:30,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:30,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:33,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:33,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:33,019 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:36,029 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:36,035 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:36,035 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:39,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:39,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:39,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:42,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:42,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:42,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:45,086 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.41 % 2021-12-16 13:03:45,092 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:45,092 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:48,102 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.61 % 2021-12-16 13:03:48,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:48,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:51,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.80 % 2021-12-16 13:03:51,122 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:51,122 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:54,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.80 % 2021-12-16 13:03:54,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:54,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:03:57,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.80 % 2021-12-16 13:03:57,165 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:03:57,165 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:00,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.80 % 2021-12-16 13:04:00,188 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:00,188 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:03,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 65.80 % 2021-12-16 13:04:03,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:03,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:06,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.19 % 2021-12-16 13:04:06,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:06,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:09,227 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.19 % 2021-12-16 13:04:09,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:09,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:12,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.19 % 2021-12-16 13:04:12,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:12,259 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:15,269 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:15,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:15,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:18,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:18,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:18,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:21,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:21,311 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:21,311 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:24,321 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:24,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:24,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:27,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:27,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:27,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:30,354 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:30,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:30,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:33,372 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:33,387 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:33,387 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:36,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:36,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:36,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:39,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:39,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:39,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:42,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:42,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:42,436 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:45,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:45,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:45,454 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:48,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:48,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:48,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:51,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.39 % 2021-12-16 13:04:51,497 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:51,497 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:54,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:04:54,512 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:54,513 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:04:57,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:04:57,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:04:57,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:00,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:05:00,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:00,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:03,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:05:03,572 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:03,572 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:06,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:05:06,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:06,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:09,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:05:09,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:09,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:12,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.58 % 2021-12-16 13:05:12,625 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:12,625 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:15,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.78 % 2021-12-16 13:05:15,640 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:15,640 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:18,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.78 % 2021-12-16 13:05:18,660 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:18,660 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:21,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:21,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:21,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:24,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:24,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:24,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:27,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:27,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:27,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:30,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:30,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:30,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:33,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:33,752 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:33,752 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:36,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 66.97 % 2021-12-16 13:05:36,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:36,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:39,798 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.37 % 2021-12-16 13:05:39,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:39,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:42,819 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.37 % 2021-12-16 13:05:42,823 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:42,823 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:45,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.37 % 2021-12-16 13:05:45,838 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:45,838 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:48,848 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.37 % 2021-12-16 13:05:48,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:48,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:51,867 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:05:51,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:51,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:54,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:05:54,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:54,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:05:57,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:05:57,911 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:05:57,911 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:00,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:00,924 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:00,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:03,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:03,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:03,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:06,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:06,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:06,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:09,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:09,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:09,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:12,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:12,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:12,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:16,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:16,012 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:16,012 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:19,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:19,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:19,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:22,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:22,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:22,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:25,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:25,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:25,061 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:28,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.56 % 2021-12-16 13:06:28,079 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:28,081 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:31,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.76 % 2021-12-16 13:06:31,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:31,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:34,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.76 % 2021-12-16 13:06:34,109 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:34,109 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:37,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.95 % 2021-12-16 13:06:37,127 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:37,127 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:40,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.95 % 2021-12-16 13:06:40,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:40,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:43,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 67.95 % 2021-12-16 13:06:43,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:43,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:46,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.15 % 2021-12-16 13:06:46,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:46,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:49,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.15 % 2021-12-16 13:06:49,193 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:49,194 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:52,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.15 % 2021-12-16 13:06:52,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:52,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:55,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.34 % 2021-12-16 13:06:55,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:55,237 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:06:58,252 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:06:58,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:06:58,259 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:01,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:01,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:01,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:04,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:04,308 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:04,308 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:07,316 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:07,322 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:07,322 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:10,332 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:10,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:10,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:13,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:13,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:13,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:16,387 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:16,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:16,400 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:19,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:19,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:19,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:22,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:22,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:22,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:25,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:25,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:25,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:28,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:28,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:28,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:31,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:31,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:31,476 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:34,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.54 % 2021-12-16 13:07:34,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:34,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:37,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:37,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:37,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:40,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:40,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:40,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:43,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:43,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:43,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:46,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:46,579 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:46,579 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:49,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:49,602 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:49,602 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:52,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:52,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:52,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:55,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.73 % 2021-12-16 13:07:55,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:55,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:07:58,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.93 % 2021-12-16 13:07:58,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:07:58,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:01,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.93 % 2021-12-16 13:08:01,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:01,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:04,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 68.93 % 2021-12-16 13:08:04,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:04,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:07,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.12 % 2021-12-16 13:08:07,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:07,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:10,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.32 % 2021-12-16 13:08:10,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:10,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:13,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.51 % 2021-12-16 13:08:13,742 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:13,742 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:16,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.51 % 2021-12-16 13:08:16,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:16,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:19,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:19,785 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:19,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:22,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:22,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:22,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:25,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:25,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:25,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:28,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:28,859 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:28,859 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:31,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:31,875 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:31,875 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:34,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:34,891 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:34,891 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:37,900 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:37,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:37,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:40,920 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:40,928 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:40,928 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:43,943 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:43,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:43,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:46,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:46,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:46,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:49,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:49,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:49,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:52,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:53,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:53,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:56,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:56,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:56,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:08:59,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:08:59,046 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:08:59,046 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:02,061 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:02,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:02,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:05,079 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:05,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:05,085 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:08,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:08,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:08,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:11,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:11,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:11,122 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:14,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:14,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:14,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:17,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.71 % 2021-12-16 13:09:17,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:17,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:20,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.91 % 2021-12-16 13:09:20,167 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:20,167 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:23,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.91 % 2021-12-16 13:09:23,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:23,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:26,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 69.91 % 2021-12-16 13:09:26,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:26,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:29,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.10 % 2021-12-16 13:09:29,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:29,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:32,249 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.10 % 2021-12-16 13:09:32,255 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:32,256 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:35,265 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.30 % 2021-12-16 13:09:35,272 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:35,272 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:38,285 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.69 % 2021-12-16 13:09:38,292 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:38,292 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:41,302 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.69 % 2021-12-16 13:09:41,312 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:41,312 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:44,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:44,331 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:44,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:47,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:47,348 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:47,348 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:50,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:50,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:50,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:53,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:53,375 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:53,375 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:56,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:56,391 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:56,391 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:09:59,401 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:09:59,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:09:59,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:02,426 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:02,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:02,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:05,448 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:05,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:05,457 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:08,469 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:08,474 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:08,474 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:11,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:11,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:11,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:14,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:14,513 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:14,513 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:17,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:17,535 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:17,535 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:20,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:20,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:20,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:23,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:23,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:23,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:26,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:26,584 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:26,584 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:29,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:29,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:29,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:32,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:32,614 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:32,615 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:35,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:35,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:35,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:38,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:38,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:38,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:41,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:41,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:41,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:44,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 70.88 % 2021-12-16 13:10:44,686 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:44,686 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:47,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.08 % 2021-12-16 13:10:47,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:47,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:50,719 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.08 % 2021-12-16 13:10:50,727 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:50,727 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:53,737 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.27 % 2021-12-16 13:10:53,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:53,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:56,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.27 % 2021-12-16 13:10:56,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:56,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:10:59,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.27 % 2021-12-16 13:10:59,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:10:59,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:02,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.47 % 2021-12-16 13:11:02,795 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:02,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:05,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.66 % 2021-12-16 13:11:05,813 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:05,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:08,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.66 % 2021-12-16 13:11:08,832 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:08,832 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:11,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.66 % 2021-12-16 13:11:11,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:11,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:14,857 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 71.66 % 2021-12-16 13:11:14,862 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:14,862 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:17,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:17,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:17,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:20,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:20,897 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:20,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:23,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:23,915 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:23,917 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:26,926 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:26,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:26,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:29,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:29,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:29,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:32,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:32,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:32,961 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:35,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:35,979 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:35,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:38,989 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:38,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:38,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:42,006 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:42,015 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:42,015 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:45,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:45,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:45,029 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:48,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:48,046 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:48,046 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:51,055 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:51,061 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:51,061 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:54,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:54,079 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:54,079 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:11:57,089 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:11:57,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:11:57,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:00,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:12:00,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:00,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:03,125 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.05 % 2021-12-16 13:12:03,132 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:03,132 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:06,141 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.25 % 2021-12-16 13:12:06,147 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:06,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:09,155 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.25 % 2021-12-16 13:12:09,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:09,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:12,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.25 % 2021-12-16 13:12:12,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:12,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:15,188 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.45 % 2021-12-16 13:12:15,197 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:15,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:18,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.45 % 2021-12-16 13:12:18,216 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:18,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:21,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.45 % 2021-12-16 13:12:21,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:21,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:24,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.45 % 2021-12-16 13:12:24,249 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:24,249 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:27,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:27,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:27,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:30,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:30,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:30,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:33,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:33,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:33,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:36,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:36,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:36,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:39,334 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:39,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:39,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:42,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.64 % 2021-12-16 13:12:42,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:42,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:45,367 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 72.84 % 2021-12-16 13:12:45,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:45,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:48,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.03 % 2021-12-16 13:12:48,390 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:48,390 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:51,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:12:51,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:51,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:54,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:12:54,424 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:54,424 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:12:57,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:12:57,438 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:12:57,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:00,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:00,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:00,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:03,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:03,469 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:03,469 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:06,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:06,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:06,489 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:09,499 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:09,506 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:09,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:12,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:12,525 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:12,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:15,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:15,539 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:15,540 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:18,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:18,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:18,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:21,566 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:21,573 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:21,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:24,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:24,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:24,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:27,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.23 % 2021-12-16 13:13:27,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:27,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:30,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.42 % 2021-12-16 13:13:30,632 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:30,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:33,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.42 % 2021-12-16 13:13:33,686 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:33,687 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:36,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.42 % 2021-12-16 13:13:36,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:36,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:39,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.62 % 2021-12-16 13:13:39,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:39,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:42,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.62 % 2021-12-16 13:13:42,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:42,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:45,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.62 % 2021-12-16 13:13:45,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:45,752 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:48,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.62 % 2021-12-16 13:13:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:48,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:51,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:13:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:51,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:54,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:13:54,798 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:54,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:13:57,808 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:13:57,816 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:13:57,816 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:00,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:14:00,841 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:00,841 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:03,851 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:14:03,857 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:03,857 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:06,865 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:14:06,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:06,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:09,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 73.81 % 2021-12-16 13:14:09,887 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:09,887 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:12,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.01 % 2021-12-16 13:14:12,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:12,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:15,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.01 % 2021-12-16 13:14:15,928 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:15,928 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:18,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.01 % 2021-12-16 13:14:18,948 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:18,948 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:21,959 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.01 % 2021-12-16 13:14:21,964 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:21,964 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:24,974 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.20 % 2021-12-16 13:14:24,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:24,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:27,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.20 % 2021-12-16 13:14:27,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:27,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:31,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.20 % 2021-12-16 13:14:31,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:31,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:34,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.40 % 2021-12-16 13:14:34,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:34,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:37,049 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.40 % 2021-12-16 13:14:37,058 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:37,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:40,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.40 % 2021-12-16 13:14:40,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:40,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:43,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.40 % 2021-12-16 13:14:43,089 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:43,089 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:46,099 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.40 % 2021-12-16 13:14:46,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:46,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:49,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.59 % 2021-12-16 13:14:49,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:49,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:52,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.59 % 2021-12-16 13:14:52,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:52,151 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:55,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.59 % 2021-12-16 13:14:55,168 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:55,168 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:14:58,180 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:14:58,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:14:58,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:01,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:01,206 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:01,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:04,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:04,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:04,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:07,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:07,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:07,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:10,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:10,289 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:10,289 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:13,301 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:13,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:13,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:16,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:16,325 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:16,325 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:19,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:19,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:19,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:22,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:22,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:22,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:25,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:25,382 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:25,382 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:28,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:28,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:28,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:31,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.79 % 2021-12-16 13:15:31,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:31,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:34,421 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.98 % 2021-12-16 13:15:34,430 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:34,430 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:37,441 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 74.98 % 2021-12-16 13:15:37,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:37,450 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:40,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:40,469 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:40,470 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:43,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:43,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:43,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:46,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:46,499 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:46,499 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:49,507 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:49,515 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:49,515 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:52,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:52,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:52,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:55,543 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:55,552 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:55,552 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:15:58,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.18 % 2021-12-16 13:15:58,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:15:58,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:01,576 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.57 % 2021-12-16 13:16:01,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:01,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:04,594 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.57 % 2021-12-16 13:16:04,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:04,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:07,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.57 % 2021-12-16 13:16:07,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:07,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:10,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.57 % 2021-12-16 13:16:10,637 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:10,637 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:13,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:13,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:13,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:16,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:16,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:16,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:19,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:19,686 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:19,686 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:22,696 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:22,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:22,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:25,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:25,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:25,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:28,727 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:28,733 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:28,733 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:31,743 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:31,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:31,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:34,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:34,763 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:34,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:37,773 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.77 % 2021-12-16 13:16:37,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:37,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:40,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:40,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:40,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:43,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:43,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:43,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:46,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:46,836 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:46,836 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:49,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:49,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:49,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:52,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:52,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:52,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:55,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:55,885 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:55,885 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:16:58,895 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:16:58,903 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:16:58,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:01,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:17:01,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:01,923 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:04,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:17:04,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:04,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:07,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 75.96 % 2021-12-16 13:17:07,950 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:07,950 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:10,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:10,969 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:10,969 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:13,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:13,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:13,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:17,012 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:17,023 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:17,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:20,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:20,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:20,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:23,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:23,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:23,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:26,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.16 % 2021-12-16 13:17:26,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:26,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:29,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.35 % 2021-12-16 13:17:29,091 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:29,091 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:32,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-16 13:17:32,111 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:32,111 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:35,121 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-16 13:17:35,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:35,128 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:38,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-16 13:17:38,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:38,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:41,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.55 % 2021-12-16 13:17:41,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:41,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:44,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:44,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:47,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:47,203 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:47,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:50,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:50,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:53,232 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:53,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:53,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:56,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:56,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:56,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:17:59,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:17:59,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:17:59,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:02,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:18:02,288 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:02,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:05,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:18:05,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:05,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:08,312 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:18:08,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:08,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:11,326 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:18:11,334 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:11,334 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:14,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.74 % 2021-12-16 13:18:14,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:14,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:17,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.94 % 2021-12-16 13:18:17,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:17,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:20,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.94 % 2021-12-16 13:18:20,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:20,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:23,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 76.94 % 2021-12-16 13:18:23,411 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:23,411 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:26,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.13 % 2021-12-16 13:18:26,433 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:26,433 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:29,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.13 % 2021-12-16 13:18:29,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:29,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:32,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.13 % 2021-12-16 13:18:32,474 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:32,474 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:35,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.33 % 2021-12-16 13:18:35,493 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:35,493 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:38,503 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.33 % 2021-12-16 13:18:38,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:38,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:41,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.33 % 2021-12-16 13:18:41,523 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:41,523 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:44,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:44,540 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:44,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:47,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:47,561 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:47,561 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:50,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:50,576 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:50,576 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:53,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:53,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:53,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:56,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:56,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:56,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:18:59,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:18:59,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:18:59,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:02,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:19:02,651 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:02,651 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:05,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:19:05,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:05,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:08,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:19:08,683 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:08,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:11,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.52 % 2021-12-16 13:19:11,701 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:11,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:14,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.72 % 2021-12-16 13:19:14,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:14,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:17,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.72 % 2021-12-16 13:19:17,739 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:17,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:20,753 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.72 % 2021-12-16 13:19:20,766 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:20,766 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:23,776 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.72 % 2021-12-16 13:19:23,782 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:23,782 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:26,793 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:26,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:26,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:29,809 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:29,818 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:29,818 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:32,828 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:32,836 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:32,836 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:35,846 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:35,855 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:35,855 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:38,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:38,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:38,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:41,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:41,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:41,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:44,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:44,903 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:44,903 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:47,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:47,918 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:47,918 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:50,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 77.91 % 2021-12-16 13:19:50,937 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:50,937 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:53,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.11 % 2021-12-16 13:19:53,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:53,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:56,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.11 % 2021-12-16 13:19:56,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:56,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:19:59,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.11 % 2021-12-16 13:19:59,982 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:19:59,982 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:02,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.11 % 2021-12-16 13:20:03,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:03,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:06,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.11 % 2021-12-16 13:20:06,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:06,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:09,030 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.30 % 2021-12-16 13:20:09,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:09,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:12,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.30 % 2021-12-16 13:20:12,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:12,059 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:15,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.50 % 2021-12-16 13:20:15,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:15,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:18,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.50 % 2021-12-16 13:20:18,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:18,096 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:21,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:21,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:21,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:24,132 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:24,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:24,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:27,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:27,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:27,161 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:30,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:30,181 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:30,181 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:33,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:33,198 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:33,198 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:36,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:36,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:36,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:39,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.69 % 2021-12-16 13:20:39,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:39,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:42,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:42,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:42,265 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:45,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:45,288 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:45,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:48,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:48,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:48,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:51,323 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:51,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:51,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:54,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:54,346 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:54,346 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:20:57,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:20:57,372 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:20:57,372 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:00,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 78.89 % 2021-12-16 13:21:00,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:00,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:03,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:03,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:03,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:06,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:06,443 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:06,443 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:09,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:09,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:09,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:12,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:12,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:12,482 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:15,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:15,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:15,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:18,520 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.08 % 2021-12-16 13:21:18,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:18,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:21,548 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:21,558 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:21,558 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:24,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:24,576 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:24,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:27,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:27,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:27,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:30,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:30,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:30,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:33,634 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:33,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:33,642 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:36,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:36,663 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:36,663 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:39,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:39,687 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:39,687 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:42,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:42,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:42,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:45,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.28 % 2021-12-16 13:21:45,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:45,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:48,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:21:48,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:48,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:51,760 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:21:51,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:51,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:54,783 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:21:54,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:54,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:21:57,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:21:57,811 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:21:57,811 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:00,823 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:22:00,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:00,829 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:03,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:22:03,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:03,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:06,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:22:06,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:06,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:09,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.47 % 2021-12-16 13:22:09,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:09,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:12,911 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.87 % 2021-12-16 13:22:12,915 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:12,915 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:15,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 79.87 % 2021-12-16 13:22:15,933 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:15,933 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:18,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.06 % 2021-12-16 13:22:18,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:18,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:21,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.06 % 2021-12-16 13:22:21,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:21,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:24,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:24,999 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:24,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:28,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:28,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:28,020 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:31,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:31,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:31,039 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:34,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:34,055 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:34,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:37,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:37,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:37,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:40,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:40,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:40,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:43,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:43,122 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:43,122 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:46,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:46,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:46,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:49,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:49,160 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:49,160 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:52,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:52,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:52,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:55,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:55,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:55,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:22:58,213 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:22:58,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:22:58,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:01,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:01,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:01,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:04,255 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:04,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:04,259 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:07,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:07,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:07,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:10,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:10,303 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:10,303 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:13,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:13,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:13,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:16,349 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.25 % 2021-12-16 13:23:16,359 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:16,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:19,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.45 % 2021-12-16 13:23:19,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:19,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:22,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.45 % 2021-12-16 13:23:22,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:22,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:25,411 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.45 % 2021-12-16 13:23:25,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:25,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:28,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.45 % 2021-12-16 13:23:28,441 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:28,443 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:31,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.45 % 2021-12-16 13:23:31,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:31,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:34,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.64 % 2021-12-16 13:23:34,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:34,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:37,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:37,497 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:37,497 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:40,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:40,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:40,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:43,535 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:43,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:43,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:46,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:46,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:46,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:49,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:49,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:49,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:52,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:52,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:52,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:55,625 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:55,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:55,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:23:58,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:23:58,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:23:58,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:01,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 80.84 % 2021-12-16 13:24:01,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:01,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:04,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:04,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:04,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:07,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:07,732 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:07,732 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:10,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:10,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:10,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:13,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:13,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:13,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:16,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:16,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:16,801 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:19,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:19,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:19,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:22,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:22,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:22,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:25,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.04 % 2021-12-16 13:24:25,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:25,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:28,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.23 % 2021-12-16 13:24:28,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:28,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:31,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.23 % 2021-12-16 13:24:31,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:31,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:34,921 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.43 % 2021-12-16 13:24:34,931 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:34,931 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:37,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.43 % 2021-12-16 13:24:37,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:37,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:40,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.43 % 2021-12-16 13:24:40,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:40,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:43,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.62 % 2021-12-16 13:24:43,989 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:43,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:47,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.62 % 2021-12-16 13:24:47,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:47,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:50,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:24:50,030 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:50,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:53,045 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:24:53,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:53,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:56,065 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:24:56,071 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:56,071 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:24:59,084 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:24:59,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:24:59,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:02,101 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:02,107 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:02,107 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:05,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:05,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:05,134 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:08,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:08,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:08,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:11,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:11,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:11,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:14,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:14,191 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:14,191 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:17,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:17,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:17,214 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:20,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:20,237 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:20,237 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:23,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:23,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:23,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:26,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:26,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:26,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:29,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:29,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:29,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:32,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:32,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:32,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:35,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:35,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:35,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:38,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:38,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:38,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:41,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 81.82 % 2021-12-16 13:25:41,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:41,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:44,415 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:44,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:44,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:47,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:47,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:47,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:50,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:50,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:50,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:53,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:53,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:53,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:56,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:56,523 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:56,523 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:25:59,533 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:25:59,539 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:25:59,539 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:02,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:26:02,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:02,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:05,568 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:26:05,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:05,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:08,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:26:08,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:08,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:11,609 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.01 % 2021-12-16 13:26:11,617 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:11,617 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:14,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:14,632 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:14,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:17,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:17,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:17,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:20,661 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:20,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:20,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:23,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:23,687 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:23,687 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:26,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:26,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:26,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:29,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:29,724 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:29,724 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:32,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:32,740 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:32,740 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:35,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.21 % 2021-12-16 13:26:35,759 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:35,759 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:38,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.40 % 2021-12-16 13:26:38,779 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:38,779 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:41,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.40 % 2021-12-16 13:26:41,801 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:41,801 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:44,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:44,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:44,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:47,839 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:47,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:47,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:50,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:50,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:50,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:53,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:53,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:53,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:56,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:56,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:56,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:26:59,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.60 % 2021-12-16 13:26:59,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:26:59,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:02,938 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.79 % 2021-12-16 13:27:02,944 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:02,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:05,956 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.79 % 2021-12-16 13:27:05,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:05,961 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:08,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.79 % 2021-12-16 13:27:08,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:08,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:11,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:12,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:12,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:15,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:15,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:15,020 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:18,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:18,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:18,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:21,052 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:21,058 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:21,058 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:24,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:24,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:24,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:27,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:27,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:27,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:30,115 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:30,124 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:30,124 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:33,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:33,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:33,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:36,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:36,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:36,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:39,164 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:39,176 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:39,176 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:42,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:42,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:42,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:45,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:45,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:45,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:48,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:48,234 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:48,234 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:51,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:51,249 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:51,249 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:54,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:54,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:54,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:27:57,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:27:57,283 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:27:57,283 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:00,293 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:28:00,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:00,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:03,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:28:03,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:03,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:06,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 82.99 % 2021-12-16 13:28:06,334 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:06,334 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:09,345 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.18 % 2021-12-16 13:28:09,352 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:09,352 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:12,362 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.18 % 2021-12-16 13:28:12,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:12,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:15,381 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.18 % 2021-12-16 13:28:15,387 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:15,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:18,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.18 % 2021-12-16 13:28:18,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:18,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:21,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.18 % 2021-12-16 13:28:21,420 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:21,420 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:24,430 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:24,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:24,436 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:27,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:27,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:27,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:30,464 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:30,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:30,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:33,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:33,489 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:33,490 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:36,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:36,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:36,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:39,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:39,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:39,530 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:42,540 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:42,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:42,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:45,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:45,572 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:45,572 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:48,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:48,594 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:48,594 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:51,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:51,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:51,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:54,618 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:54,627 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:54,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:28:57,637 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:28:57,644 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:28:57,644 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:00,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.38 % 2021-12-16 13:29:00,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:00,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:03,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.58 % 2021-12-16 13:29:03,683 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:03,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:06,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.77 % 2021-12-16 13:29:06,698 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:06,698 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:09,709 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.77 % 2021-12-16 13:29:09,714 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:09,714 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:12,724 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:12,732 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:12,732 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:15,742 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:15,750 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:15,750 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:18,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:18,767 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:18,767 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:21,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:21,789 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:21,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:24,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:24,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:24,806 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:27,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:27,821 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:27,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:30,834 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:30,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:30,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:33,854 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 83.97 % 2021-12-16 13:29:33,862 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:33,862 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:36,872 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:36,878 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:36,880 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:39,890 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:39,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:39,894 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:42,905 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:42,911 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:42,911 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:45,933 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:45,944 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:45,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:48,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:48,966 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:48,966 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:51,976 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:51,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:51,984 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:54,994 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:55,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:55,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:29:58,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:29:58,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:29:58,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:01,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:01,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:01,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:04,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:04,053 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:04,053 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:07,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:07,072 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:07,072 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:10,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:10,088 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:10,088 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:13,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:13,102 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:13,102 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:16,114 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:16,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:16,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:19,131 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:19,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:19,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:22,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:22,157 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:22,157 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:25,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.16 % 2021-12-16 13:30:25,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:25,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:28,184 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.36 % 2021-12-16 13:30:28,188 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:28,188 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:31,198 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.36 % 2021-12-16 13:30:31,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:31,204 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:34,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.36 % 2021-12-16 13:30:34,227 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:34,227 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:37,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:37,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:37,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:40,260 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:40,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:40,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:43,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:43,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:43,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:46,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:46,298 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:46,298 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:49,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:49,316 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:49,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:52,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:52,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:52,341 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:55,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:55,358 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:55,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:30:58,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:30:58,372 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:30:58,374 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:01,384 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:31:01,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:01,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:04,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:31:04,408 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:04,408 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:07,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:31:07,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:07,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:10,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:31:10,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:10,447 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:13,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.55 % 2021-12-16 13:31:13,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:13,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:16,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.75 % 2021-12-16 13:31:16,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:16,477 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:19,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.94 % 2021-12-16 13:31:19,516 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:19,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:22,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 84.94 % 2021-12-16 13:31:22,540 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:22,540 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:25,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:25,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:25,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:28,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:28,579 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:28,579 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:31,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:31,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:31,595 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:34,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:34,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:34,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:37,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:37,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:37,635 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:40,645 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:40,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:40,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:43,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:43,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:43,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:46,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:46,690 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:46,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:49,700 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.14 % 2021-12-16 13:31:49,706 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:49,706 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:52,717 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:31:52,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:52,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:55,749 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:31:55,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:55,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:31:58,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:31:58,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:31:58,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:01,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:01,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:01,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:04,801 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:04,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:04,806 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:07,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:07,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:07,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:10,832 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:10,839 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:10,839 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:13,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:13,877 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:13,877 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:16,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:16,897 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:16,897 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:19,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:19,913 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:19,913 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:22,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:22,930 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:22,930 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:25,940 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.34 % 2021-12-16 13:32:25,946 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:25,946 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:28,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:28,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:28,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:31,977 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:31,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:31,986 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:34,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:35,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:35,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:38,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:38,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:38,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:41,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:41,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:41,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:44,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:44,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:44,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:47,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:47,084 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:47,084 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:50,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:50,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:50,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:53,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:53,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:53,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:56,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:56,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:56,158 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:32:59,168 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:32:59,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:32:59,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:02,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:33:02,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:02,197 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:05,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.53 % 2021-12-16 13:33:05,219 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:05,219 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:08,229 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.73 % 2021-12-16 13:33:08,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:08,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:11,244 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.73 % 2021-12-16 13:33:11,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:11,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:14,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.73 % 2021-12-16 13:33:14,265 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:14,265 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:17,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:17,288 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:17,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:20,298 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:20,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:20,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:23,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:23,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:23,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:26,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:26,341 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:26,341 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:29,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:29,359 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:29,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:32,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 85.92 % 2021-12-16 13:33:32,381 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:32,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:35,392 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.12 % 2021-12-16 13:33:35,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:35,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:38,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.12 % 2021-12-16 13:33:38,418 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:38,418 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:41,433 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:41,437 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:41,437 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:44,447 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:44,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:44,456 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:47,466 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:47,470 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:47,470 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:50,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:50,505 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:50,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:53,516 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:53,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:53,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:56,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:56,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:56,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:33:59,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:33:59,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:33:59,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:02,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:34:02,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:02,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:05,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.31 % 2021-12-16 13:34:05,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:05,594 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:08,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:08,615 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:08,617 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:11,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:11,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:11,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:14,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:14,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:14,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:17,663 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:17,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:17,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:20,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:20,690 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:20,690 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:23,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.51 % 2021-12-16 13:34:23,711 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:23,711 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:26,723 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:26,732 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:26,732 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:29,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:29,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:29,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:32,763 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:32,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:32,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:35,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:35,785 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:35,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:38,795 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:38,803 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:38,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:41,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:41,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:41,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:44,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:44,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:44,844 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:47,852 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:47,858 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:47,858 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:50,868 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:50,872 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:50,874 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:53,887 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:53,894 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:53,895 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:56,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:56,917 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:56,917 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:34:59,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:34:59,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:34:59,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:02,966 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:35:02,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:02,974 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:05,984 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:35:05,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:05,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:09,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:35:09,020 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:09,020 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:12,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:35:12,039 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:12,040 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:15,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.70 % 2021-12-16 13:35:15,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:15,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:18,072 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.90 % 2021-12-16 13:35:18,081 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:18,081 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:21,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 86.90 % 2021-12-16 13:35:21,096 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:21,098 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:24,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:24,114 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:24,114 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:27,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:27,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:27,137 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:30,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:30,158 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:30,158 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:33,170 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:33,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:33,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:36,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:36,193 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:36,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:39,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:39,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:39,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:42,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:42,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:42,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:45,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:45,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:45,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:48,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:48,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:48,259 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:51,270 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.09 % 2021-12-16 13:35:51,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:51,279 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:54,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.29 % 2021-12-16 13:35:54,293 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:54,295 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:35:57,303 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.29 % 2021-12-16 13:35:57,308 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:35:57,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:00,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.29 % 2021-12-16 13:36:00,326 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:00,326 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:03,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.29 % 2021-12-16 13:36:03,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:03,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:06,352 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:06,361 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:06,361 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:09,371 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:09,377 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:09,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:12,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:12,392 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:12,392 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:15,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:15,408 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:15,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:18,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:18,426 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:18,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:21,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:21,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:21,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:24,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:24,461 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:24,461 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:27,471 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.49 % 2021-12-16 13:36:27,476 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:27,477 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:30,486 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.68 % 2021-12-16 13:36:30,492 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:30,492 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:33,502 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:33,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:33,509 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:36,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:36,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:36,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:39,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:39,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:39,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:42,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:42,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:42,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:45,588 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:45,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:45,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:48,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:48,608 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:48,608 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:51,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:51,628 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:51,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:54,640 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:54,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:54,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:36:57,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:36:57,667 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:36:57,668 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:00,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:00,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:00,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:03,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:03,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:03,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:06,710 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:06,717 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:06,717 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:09,729 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:09,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:09,734 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:12,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:12,755 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:12,755 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:15,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:15,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:15,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:18,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 87.88 % 2021-12-16 13:37:18,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:18,792 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:21,803 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:21,811 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:21,811 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:24,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:24,829 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:24,829 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:27,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:27,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:27,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:30,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:30,869 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:30,869 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:33,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:33,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:33,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:36,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:36,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:36,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:39,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.07 % 2021-12-16 13:37:39,917 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:39,917 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:42,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:42,933 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:42,933 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:45,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:45,957 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:45,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:48,969 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:48,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:48,977 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:51,986 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:51,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:51,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:55,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:55,010 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:55,010 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:37:58,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:37:58,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:37:58,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:01,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:01,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:01,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:04,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:04,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:04,076 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:07,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:07,096 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:07,096 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:10,108 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:10,117 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:10,117 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:13,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:13,137 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:13,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:16,148 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.27 % 2021-12-16 13:38:16,155 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:16,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:19,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.46 % 2021-12-16 13:38:19,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:19,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:22,209 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.46 % 2021-12-16 13:38:22,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:22,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:25,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.46 % 2021-12-16 13:38:25,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:25,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:28,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.66 % 2021-12-16 13:38:28,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:28,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:31,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.66 % 2021-12-16 13:38:31,279 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:31,280 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:34,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:34,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:34,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:37,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:37,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:37,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:40,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:40,351 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:40,351 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:43,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:43,367 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:43,367 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:46,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:46,381 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:46,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:49,394 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 88.86 % 2021-12-16 13:38:49,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:49,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:52,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:38:52,426 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:52,426 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:55,437 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:38:55,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:55,447 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:38:58,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:38:58,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:38:58,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:01,479 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:01,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:01,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:04,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:04,499 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:04,499 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:07,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:07,515 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:07,515 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:10,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:10,532 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:10,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:13,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:13,552 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:13,552 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:16,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:16,569 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:16,569 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:19,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:19,585 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:19,585 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:22,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:22,599 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:22,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:25,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:25,621 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:25,622 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:28,632 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.05 % 2021-12-16 13:39:28,641 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:28,641 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:31,651 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:31,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:31,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:34,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:34,673 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:34,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:37,683 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:37,688 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:37,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:40,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:40,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:40,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:43,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:43,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:43,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:46,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:46,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:46,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:49,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:49,770 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:49,770 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:52,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:52,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:52,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:55,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:55,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:55,806 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:39:58,821 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:39:58,831 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:39:58,831 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:01,842 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:40:01,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:01,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:04,862 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:40:04,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:04,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:07,877 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.25 % 2021-12-16 13:40:07,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:07,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:10,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:10,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:10,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:13,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:13,959 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:13,959 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:16,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:16,997 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:16,999 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:20,007 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:20,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:20,013 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:23,025 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:23,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:23,029 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:26,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:26,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:26,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:29,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:29,068 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:29,068 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:32,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:32,086 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:32,086 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:35,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.44 % 2021-12-16 13:40:35,102 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:35,102 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:38,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.64 % 2021-12-16 13:40:38,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:38,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:41,128 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.64 % 2021-12-16 13:40:41,134 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:41,135 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:44,144 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.83 % 2021-12-16 13:40:44,151 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:44,153 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:47,161 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.83 % 2021-12-16 13:40:47,170 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:47,170 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:50,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 89.83 % 2021-12-16 13:40:50,184 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:50,184 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:53,194 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.03 % 2021-12-16 13:40:53,200 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:53,200 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:56,210 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.03 % 2021-12-16 13:40:56,214 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:56,214 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:40:59,236 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:40:59,246 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:40:59,246 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:02,256 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:02,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:02,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:05,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:05,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:05,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:08,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:08,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:08,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:11,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:11,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:11,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:14,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:14,334 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:14,334 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:17,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:17,367 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:17,367 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:20,378 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:20,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:20,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:23,398 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:23,404 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:23,404 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:26,414 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:26,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:26,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:29,434 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.22 % 2021-12-16 13:41:29,438 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:29,438 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:32,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:32,464 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:32,464 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:35,474 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:35,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:35,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:38,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:38,502 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:38,502 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:41,513 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:41,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:41,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:44,532 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:44,542 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:44,542 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:47,551 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:47,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:47,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:50,572 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:50,578 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:50,578 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:53,589 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:53,598 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:53,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:56,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:56,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:56,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:41:59,628 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:41:59,637 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:41:59,637 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:02,647 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:02,655 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:02,655 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:05,665 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:05,671 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:05,671 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:08,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:08,686 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:08,686 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:11,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:11,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:11,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:14,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:14,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:14,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:17,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:17,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:17,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:20,759 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:20,765 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:20,765 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:23,775 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:23,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:23,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:26,792 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:26,801 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:26,801 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:29,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:29,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:29,826 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:32,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:32,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:32,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:35,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:35,864 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:35,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:38,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.42 % 2021-12-16 13:42:38,885 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:38,887 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:41,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.61 % 2021-12-16 13:42:41,907 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:41,907 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:44,918 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.61 % 2021-12-16 13:42:44,926 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:44,926 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:47,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.81 % 2021-12-16 13:42:47,947 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:47,947 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:50,957 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.81 % 2021-12-16 13:42:50,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:50,963 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:53,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.81 % 2021-12-16 13:42:53,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:53,980 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:42:56,990 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.81 % 2021-12-16 13:42:56,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:42:56,996 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:00,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 90.81 % 2021-12-16 13:43:00,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:00,016 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:03,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.01 % 2021-12-16 13:43:03,033 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:03,033 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:06,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.01 % 2021-12-16 13:43:06,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:06,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:09,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.01 % 2021-12-16 13:43:09,063 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:09,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:12,076 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.20 % 2021-12-16 13:43:12,086 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:12,086 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:15,095 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:15,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:15,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:18,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:18,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:18,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:21,140 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:21,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:21,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:24,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:24,165 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:24,165 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:27,177 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:27,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:27,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:30,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:30,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:30,204 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:33,216 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.40 % 2021-12-16 13:43:33,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:33,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:36,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:36,243 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:36,243 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:39,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:39,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:39,260 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:42,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:42,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:42,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:45,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:45,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:45,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:48,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:48,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:48,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:51,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:51,339 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:51,339 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:54,355 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:54,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:54,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:43:57,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:43:57,381 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:43:57,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:00,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:00,395 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:00,395 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:03,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:03,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:03,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:06,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:06,431 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:06,431 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:09,443 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:09,450 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:09,450 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:12,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:12,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:12,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:15,480 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:15,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:15,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:18,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:18,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:18,502 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:21,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:21,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:21,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:24,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:24,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:24,538 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:27,549 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:27,553 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:27,553 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:30,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:30,568 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:30,568 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:33,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:33,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:33,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:36,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:36,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:36,609 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:39,619 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:39,628 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:39,628 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:42,638 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:42,647 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:42,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:45,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:45,665 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:45,667 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:48,677 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.59 % 2021-12-16 13:44:48,683 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:48,683 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:51,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.79 % 2021-12-16 13:44:51,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:51,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:54,711 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.79 % 2021-12-16 13:44:54,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:54,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:44:57,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.79 % 2021-12-16 13:44:57,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:44:57,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:00,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.79 % 2021-12-16 13:45:00,778 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:00,779 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:03,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:03,793 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:03,793 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:06,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:06,829 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:06,831 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:09,841 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:09,848 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:09,848 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:12,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:12,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:12,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:15,880 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:15,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:15,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:18,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:18,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:18,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:21,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:21,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:21,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:24,931 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:24,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:24,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:27,948 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 91.98 % 2021-12-16 13:45:27,956 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:27,956 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:30,964 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.18 % 2021-12-16 13:45:30,973 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:30,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:33,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.38 % 2021-12-16 13:45:33,987 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:33,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:36,999 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.57 % 2021-12-16 13:45:37,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:37,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:40,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:40,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:40,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:43,033 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:43,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:43,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:46,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:46,066 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:46,068 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:49,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:49,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:49,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:52,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:52,099 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:52,099 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:55,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:55,115 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:55,115 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:45:58,138 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:45:58,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:45:58,145 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:01,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:01,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:01,163 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:04,173 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:04,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:04,177 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:07,187 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:07,193 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:07,193 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:10,203 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:10,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:10,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:13,221 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:13,229 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:13,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:16,239 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:16,247 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:16,247 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:19,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:19,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:19,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:22,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:22,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:22,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:25,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:25,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:25,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:28,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:28,316 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:28,316 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:31,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:31,338 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:31,338 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:34,348 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:34,354 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:34,355 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:37,365 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:37,371 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:37,371 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:40,382 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:40,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:40,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:43,400 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:43,410 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:43,410 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:46,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:46,431 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:46,431 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:49,440 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:49,446 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:49,446 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:52,454 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:52,460 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:52,460 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:55,470 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:55,479 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:55,479 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:46:58,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:46:58,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:46:58,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:01,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.77 % 2021-12-16 13:47:01,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:01,512 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:04,522 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:04,526 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:04,526 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:07,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:07,543 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:07,543 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:10,553 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:10,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:10,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:13,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:13,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:13,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:16,586 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:16,592 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:16,592 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:19,604 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 92.96 % 2021-12-16 13:47:19,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:19,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:22,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:22,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:22,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:25,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:25,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:25,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:28,671 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:28,681 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:28,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:31,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:31,701 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:31,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:34,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:34,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:34,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:37,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:37,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:37,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:40,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:40,753 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:40,753 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:43,765 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:43,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:43,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:46,785 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:46,792 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:46,793 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:49,802 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.16 % 2021-12-16 13:47:49,811 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:49,811 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:52,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.35 % 2021-12-16 13:47:52,826 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:52,826 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:55,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.35 % 2021-12-16 13:47:55,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:55,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:47:58,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:47:58,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:47:58,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:01,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:01,881 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:01,881 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:04,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:04,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:04,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:07,908 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:07,915 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:07,915 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:10,927 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:10,934 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:10,934 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:13,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:13,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:13,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:16,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:16,967 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:16,967 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:19,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:19,987 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:19,989 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:23,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:23,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:23,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:26,039 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:26,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:26,048 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:29,059 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:29,069 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:29,069 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:32,078 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:32,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:32,085 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:35,096 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:35,108 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:35,108 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:38,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:38,124 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:38,124 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:41,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:41,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:41,142 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:44,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:44,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:44,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:47,174 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:47,181 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:47,181 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:50,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:50,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:50,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:53,206 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:53,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:53,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:56,224 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:56,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:56,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:48:59,243 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:48:59,253 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:48:59,253 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:02,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:49:02,269 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:02,269 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:05,279 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:49:05,285 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:05,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:08,295 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:49:08,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:08,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:11,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:49:11,318 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:11,318 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:14,328 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 93.94 % 2021-12-16 13:49:14,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:14,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:17,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:17,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:17,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:20,374 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:20,381 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:20,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:23,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:23,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:23,400 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:26,408 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:26,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:26,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:29,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:29,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:29,436 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:32,446 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:32,454 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:32,454 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:35,463 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:35,469 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:35,469 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:38,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.13 % 2021-12-16 13:49:38,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:38,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:41,493 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.33 % 2021-12-16 13:49:41,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:41,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:44,510 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.33 % 2021-12-16 13:49:44,517 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:44,517 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:47,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.33 % 2021-12-16 13:49:47,533 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:47,533 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:50,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:49:50,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:50,551 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:53,559 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:49:53,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:53,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:56,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:49:56,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:56,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:49:59,592 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:49:59,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:49:59,598 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:02,607 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:50:02,611 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:02,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:05,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:50:05,644 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:05,644 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:08,654 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:50:08,661 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:08,661 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:11,673 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:50:11,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:11,680 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:14,694 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.53 % 2021-12-16 13:50:14,700 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:14,700 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:17,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.72 % 2021-12-16 13:50:17,721 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:17,721 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:20,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.72 % 2021-12-16 13:50:20,736 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:20,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:23,746 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.72 % 2021-12-16 13:50:23,757 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:23,757 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:26,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 94.92 % 2021-12-16 13:50:26,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:26,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:29,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:29,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:29,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:32,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:32,812 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:32,813 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:35,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:35,828 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:35,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:38,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:38,842 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:38,842 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:41,858 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:41,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:41,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:44,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:44,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:44,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:47,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:47,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:47,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:50,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:50,923 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:50,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:53,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:53,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:53,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:56,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:56,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:56,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:50:59,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:50:59,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:50:59,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:02,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:02,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:02,992 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:06,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:06,009 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:06,009 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:09,019 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:09,026 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:09,026 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:12,035 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:12,040 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:12,040 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:15,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:15,056 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:15,056 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:18,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:18,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:18,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:21,091 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:21,099 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:21,099 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:24,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:24,121 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:24,121 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:27,134 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:27,140 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:27,140 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:30,150 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:30,155 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:30,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:33,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:33,174 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:33,174 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:36,183 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:36,190 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:36,190 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:39,200 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:39,209 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:39,209 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:42,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:42,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:42,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:45,233 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:45,237 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:45,237 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:48,250 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:48,257 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:48,257 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:51,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:51,280 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:51,280 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:54,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:54,305 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:54,305 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:51:57,315 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.11 % 2021-12-16 13:51:57,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:51:57,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:00,329 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:00,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:00,335 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:03,344 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:03,348 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:03,349 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:06,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:06,368 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:06,368 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:09,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:09,384 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:09,385 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:12,395 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:12,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:12,401 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:15,410 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:15,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:15,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:18,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:18,428 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:18,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:21,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:21,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:21,447 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:24,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:24,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:24,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:27,477 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:27,482 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:27,483 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:30,490 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:30,496 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:30,496 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:33,506 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:33,510 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:33,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:36,523 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:36,530 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:36,532 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:39,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.31 % 2021-12-16 13:52:39,559 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:39,559 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:42,569 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.50 % 2021-12-16 13:52:42,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:42,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:45,584 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.50 % 2021-12-16 13:52:45,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:45,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:48,598 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.50 % 2021-12-16 13:52:48,604 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:48,604 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:51,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:52:51,619 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:51,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:54,630 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:52:54,638 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:54,638 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:52:57,648 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:52:57,654 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:52:57,654 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:00,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:00,670 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:00,670 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:03,680 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:03,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:03,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:06,697 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:06,704 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:06,704 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:09,716 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:09,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:09,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:12,736 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:12,742 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:12,742 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:15,750 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:15,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:15,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:18,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:18,770 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:18,770 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:21,779 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:21,786 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:21,786 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:24,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:24,806 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:24,806 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:27,816 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:27,822 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:27,822 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:30,835 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:30,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:30,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:33,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:33,864 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:33,864 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:36,875 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:36,884 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:36,884 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:39,892 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:39,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:39,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:42,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:42,918 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:42,918 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:45,928 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:45,934 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:45,934 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:48,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:48,951 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:48,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:51,961 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:51,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:51,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:54,980 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:54,990 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:54,990 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:53:58,000 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:53:58,005 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:53:58,005 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:01,016 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:54:01,022 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:01,023 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:04,032 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:54:04,038 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:04,038 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:07,046 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:54:07,058 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:07,058 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:10,068 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 95.89 % 2021-12-16 13:54:10,076 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:10,076 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:13,098 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:13,102 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:13,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:16,112 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:16,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:16,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:19,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:19,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:19,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:22,145 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:22,157 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:22,157 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:25,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:25,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:25,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:28,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:28,201 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:28,203 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:31,211 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:31,217 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:31,217 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:34,226 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:34,233 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:34,233 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:37,242 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:37,252 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:37,252 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:40,263 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:40,270 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:40,270 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:43,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:43,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:43,286 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:46,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:46,301 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:46,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:49,311 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:49,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:49,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:52,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:52,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:52,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:55,342 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:55,349 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:55,349 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:54:58,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:54:58,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:54:58,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:01,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:55:01,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:01,380 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:04,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:55:04,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:04,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:07,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.09 % 2021-12-16 13:55:07,414 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:07,414 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:10,424 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.48 % 2021-12-16 13:55:10,433 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:10,433 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:13,453 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.48 % 2021-12-16 13:55:13,463 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:13,463 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:16,473 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:16,477 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:16,477 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:19,497 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:19,503 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:19,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:22,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:22,517 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:22,517 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:25,526 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:25,535 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:25,535 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:28,545 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:28,555 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:28,555 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:31,565 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:31,571 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:31,571 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:34,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:34,586 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:34,586 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:37,596 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:37,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:37,602 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:40,612 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:40,619 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:40,621 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:43,631 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:43,642 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:43,644 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:46,655 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:46,665 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:46,665 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:49,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:49,680 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:49,681 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:52,691 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:52,697 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:52,697 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:55,707 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:55,716 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:55,716 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:55:58,726 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:55:58,734 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:55:58,736 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:01,744 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:01,755 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:01,755 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:04,766 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:04,773 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:04,773 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:07,782 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:07,788 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:07,788 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:10,796 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:10,802 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:10,802 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:13,815 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:13,825 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:13,825 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:16,836 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:16,845 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:16,845 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:19,855 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:19,861 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:19,861 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:22,871 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:22,878 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:22,878 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:25,888 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:25,895 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:25,895 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:28,904 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:28,914 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:28,914 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:31,924 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:31,931 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:31,931 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:34,946 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.67 % 2021-12-16 13:56:34,950 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:34,951 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:37,960 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:37,964 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:37,964 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:40,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:40,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:40,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:43,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:44,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:44,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:47,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:47,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:47,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:50,028 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:50,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:50,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:53,043 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:53,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:53,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:56,062 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:56,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:56,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:56:59,092 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:56:59,104 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:56:59,104 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:02,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:02,131 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:02,131 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:05,142 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:05,148 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:05,148 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:08,160 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:08,168 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:08,168 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:11,178 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:11,187 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:11,187 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:14,197 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:14,207 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:14,207 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:17,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:17,223 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:17,223 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:20,234 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:20,242 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:20,242 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:23,253 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:23,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:23,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:26,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:26,280 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:26,280 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:29,289 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 96.87 % 2021-12-16 13:57:29,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:29,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:32,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.06 % 2021-12-16 13:57:32,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:32,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:35,319 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.06 % 2021-12-16 13:57:35,323 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:35,323 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:38,335 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.26 % 2021-12-16 13:57:38,344 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:38,344 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:41,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:41,378 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:41,378 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:44,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:44,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:44,394 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:47,403 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:47,407 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:47,407 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:50,417 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:50,421 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:50,421 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:53,431 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:53,440 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:53,440 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:56,450 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:56,457 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:56,459 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:57:59,467 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:57:59,473 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:57:59,473 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:02,483 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:02,487 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:02,487 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:05,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:05,502 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:05,503 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:08,512 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:08,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:08,519 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:11,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:11,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:11,538 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:14,546 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:14,551 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:14,552 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:17,561 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:17,569 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:17,569 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:20,578 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:20,586 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:20,586 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:23,595 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:23,601 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:23,601 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:26,611 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:26,618 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:26,618 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:29,627 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:29,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:29,632 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:32,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:32,653 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:32,653 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:35,664 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:35,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:35,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:38,684 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:38,694 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:38,694 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:41,703 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:41,710 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:41,710 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:44,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:44,726 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:44,726 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:47,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:47,747 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:47,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:50,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.45 % 2021-12-16 13:58:50,762 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:50,763 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:53,772 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:58:53,780 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:53,780 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:56,790 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:58:56,799 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:56,799 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:58:59,812 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:58:59,819 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:58:59,819 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:02,831 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:02,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:02,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:05,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:05,849 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:05,849 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:08,864 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:08,872 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:08,872 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:11,885 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:11,898 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:11,898 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:14,907 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:14,911 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:14,911 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:17,923 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:17,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:17,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:20,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:20,943 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:20,943 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:23,951 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:23,961 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:23,961 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:26,973 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:26,983 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:26,983 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:29,993 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:30,000 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:30,000 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:33,009 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:33,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:33,015 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:36,023 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:36,030 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:36,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:39,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:39,052 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:39,052 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:42,063 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:42,071 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:42,071 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:45,082 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:45,085 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:45,086 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:48,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:48,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:48,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:51,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.65 % 2021-12-16 13:59:51,144 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:51,144 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:54,153 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 13:59:54,157 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:54,157 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 13:59:57,167 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 13:59:57,173 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 13:59:57,173 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:00,181 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 14:00:00,188 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:00,188 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:03,217 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 14:00:03,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:03,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:06,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 14:00:06,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:06,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:09,259 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 97.84 % 2021-12-16 14:00:09,263 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:09,263 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:12,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:12,280 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:12,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:15,290 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:15,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:15,299 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:18,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:18,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:18,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:21,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:21,332 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:21,332 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:24,341 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:24,346 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:24,346 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:27,361 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.04 % 2021-12-16 14:00:27,369 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:27,369 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:30,380 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:30,388 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:30,388 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:33,397 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:33,401 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:33,403 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:36,413 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:36,417 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:36,417 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:39,427 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:39,436 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:39,436 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:42,444 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:42,453 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:42,453 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:45,461 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:45,467 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:45,467 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:48,476 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:48,483 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:48,484 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:51,494 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:51,506 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:51,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:54,515 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:54,519 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:54,520 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:00:57,529 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:00:57,533 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:00:57,533 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:00,542 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:00,549 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:00,549 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:03,562 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:03,573 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:03,573 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:06,582 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:06,588 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:06,588 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:10,299 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:10,309 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:10,309 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:13,318 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:13,326 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:13,326 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:16,336 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:16,342 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:16,342 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:19,351 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:19,357 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:19,358 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:22,368 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:22,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:22,380 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:25,390 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:25,398 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:25,398 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:28,407 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:28,413 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:28,413 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:31,423 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:31,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:31,427 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:34,438 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:34,447 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:34,447 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:37,459 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:37,480 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:37,480 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:40,489 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:40,494 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:40,494 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:43,505 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:43,509 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:43,510 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:46,519 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:46,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:46,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:49,552 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:49,562 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:49,562 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:52,571 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:52,575 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:52,575 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:55,585 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:55,589 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:55,589 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:01:58,601 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:01:58,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:01:58,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:01,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:01,630 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:01,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:04,641 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:04,648 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:04,648 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:07,657 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:07,664 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:07,664 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:10,674 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:10,684 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:10,684 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:13,693 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:13,701 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:13,701 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:16,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:16,719 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:16,719 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:19,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:19,739 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:19,739 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:22,752 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:22,760 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:22,760 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:25,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:25,779 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:25,779 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:28,789 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:28,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:28,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:31,805 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:31,809 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:31,809 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:34,822 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:34,834 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:34,834 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:37,844 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:37,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:37,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:40,861 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.24 % 2021-12-16 14:02:40,868 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:40,868 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:43,878 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:43,882 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:43,882 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:46,894 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:46,901 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:46,901 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:49,917 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:49,927 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:49,927 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:52,937 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:52,944 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:52,944 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:55,953 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:55,957 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:55,957 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:02:58,967 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:02:58,971 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:02:58,973 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:01,983 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:01,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:01,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:05,002 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:05,010 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:05,010 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:08,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:08,028 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:08,028 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:11,036 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:11,042 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:11,042 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:14,051 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:14,059 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:14,061 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:17,069 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:17,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:17,078 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:20,088 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:20,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:20,095 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:23,105 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:23,109 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:23,109 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:26,118 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:26,124 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:26,124 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:29,137 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:29,142 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:29,142 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:32,154 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:32,164 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:32,164 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:35,191 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:35,198 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:35,198 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:38,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:38,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:38,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:41,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:41,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:41,230 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:44,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:44,247 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:44,249 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:47,257 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:47,266 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:47,266 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:50,276 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:50,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:50,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:53,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:53,296 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:53,296 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:56,305 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:56,311 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:56,311 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:03:59,322 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:03:59,328 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:03:59,328 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:02,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:04:02,348 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:02,348 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:05,358 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:04:05,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:05,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:08,375 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.43 % 2021-12-16 14:04:08,380 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:08,380 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:11,388 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.63 % 2021-12-16 14:04:11,394 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:11,394 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:14,436 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.63 % 2021-12-16 14:04:14,444 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:14,444 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:17,456 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.63 % 2021-12-16 14:04:17,486 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:17,486 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:20,496 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.63 % 2021-12-16 14:04:20,500 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:20,500 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:23,509 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.63 % 2021-12-16 14:04:23,515 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:23,516 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:26,525 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:26,529 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:26,529 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:29,539 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:29,546 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:29,546 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:32,556 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:32,565 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:32,565 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:35,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:35,579 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:35,581 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:38,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:38,596 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:38,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:41,608 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 98.82 % 2021-12-16 14:04:41,634 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:41,634 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:44,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:44,650 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:44,650 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:47,660 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:47,665 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:47,665 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:50,678 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:50,687 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:50,688 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:53,698 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:53,703 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:53,704 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:56,713 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:56,720 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:56,720 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:04:59,730 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:04:59,737 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:04:59,739 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:02,747 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:02,756 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:02,756 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:05,769 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:05,775 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:05,775 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:08,786 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:08,790 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:08,790 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:11,799 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:11,805 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:11,805 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:14,818 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:14,826 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:14,828 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:17,838 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:17,846 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:17,846 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:20,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:20,871 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:20,871 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:23,881 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:23,888 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:23,888 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:26,898 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:26,904 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:26,904 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:29,914 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:29,921 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:29,921 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:32,930 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:32,940 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:32,940 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:35,950 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:35,960 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:35,960 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:38,970 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:38,976 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:38,976 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:41,987 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:41,993 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:41,993 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:45,003 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:45,012 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:45,012 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:48,022 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:48,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:48,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:51,042 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:51,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:51,048 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:54,056 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:54,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:54,062 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:05:57,071 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:05:57,075 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:05:57,075 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:00,085 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:00,094 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:00,094 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:03,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:03,125 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:03,125 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:06,135 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:06,141 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:06,141 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:09,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:09,155 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:09,155 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:12,171 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:12,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:12,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:15,196 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:15,204 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:15,206 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:18,214 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:18,220 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:18,220 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:21,230 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:21,236 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:21,236 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:24,246 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:24,250 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:24,250 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:27,273 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:27,282 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:27,282 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:30,292 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:30,299 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:30,301 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:33,309 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:33,315 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:33,315 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:36,325 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:36,329 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:36,329 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:39,339 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:39,345 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:39,345 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:42,357 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:42,365 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:42,365 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:45,377 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:45,381 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:45,381 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:48,391 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:48,397 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:48,397 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:51,405 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:51,411 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:51,411 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:54,420 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:54,427 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:54,428 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:06:57,460 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.02 % 2021-12-16 14:06:57,470 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:06:57,470 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:00,484 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:00,490 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:00,490 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:03,500 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:03,506 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:03,506 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:06,517 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:06,525 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:06,525 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:09,536 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:09,545 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:09,545 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:12,558 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:12,566 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:12,566 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:15,575 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:15,582 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:15,582 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:18,591 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:18,595 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:18,596 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:21,605 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:21,609 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:21,611 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:24,621 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:24,631 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:24,631 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:27,644 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:27,657 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:27,657 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:30,667 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:30,674 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:30,674 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:33,687 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:33,693 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:33,693 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:36,701 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:36,707 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:36,707 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:39,720 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:39,727 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:39,727 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:42,739 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:42,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:42,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:45,756 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:45,762 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:45,762 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:48,770 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:48,776 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:48,776 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:51,788 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:51,796 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:51,796 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:54,806 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:54,815 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:54,815 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:07:57,825 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:07:57,835 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:07:57,835 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:00,845 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:00,851 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:00,851 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:03,859 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:03,865 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:03,865 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:06,874 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:06,880 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:06,880 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:09,891 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:09,900 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:09,900 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:12,913 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:12,923 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:12,924 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:15,934 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:15,938 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:15,938 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:18,947 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:18,953 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:18,953 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:21,963 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.22 % 2021-12-16 14:08:21,970 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:21,970 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:24,979 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.41 % 2021-12-16 14:08:24,986 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:24,987 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:27,997 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.41 % 2021-12-16 14:08:28,003 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:28,003 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:31,013 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.41 % 2021-12-16 14:08:31,017 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:31,017 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:34,026 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.61 % 2021-12-16 14:08:34,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:34,032 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:37,040 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.61 % 2021-12-16 14:08:37,048 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:37,049 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:40,058 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.61 % 2021-12-16 14:08:40,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:40,065 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:43,075 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:43,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:43,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:46,094 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:46,101 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:46,101 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:49,109 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:49,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:49,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:52,127 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:52,135 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:52,135 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:55,151 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:55,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:55,178 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:08:58,190 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:08:58,196 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:08:58,196 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:01,207 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:01,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:01,213 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:04,223 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:04,227 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:04,229 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:07,240 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:07,249 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:07,249 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:10,262 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:10,272 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:10,272 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:13,282 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:13,286 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:13,288 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:16,296 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 99.80 % 2021-12-16 14:09:16,302 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:16,302 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:19,319 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 14:09:19,319 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 14:09:24,335 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 14:09:24,336 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 14:09:27,351 | test | INFO | MainThread | [rest_client:monitorRebalance:1584] Rebalance done. Taken 10284.55 seconds to complete 2021-12-16 14:09:27,359 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1587 2021-12-16 14:09:27,359 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-12-16 14:09:32,394 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3952] Stats: {u'opsPerSec': 47394.8, u'diskFetches': 0, u'dataUsed': 10775988822581L, u'itemCount': 4981627098L, u'quotaPercentUsed': 80.42091432462136, u'storageTotals': {u'hdd': {u'usedByData': 13089975146410L, u'used': 13276012032815L, u'quotaTotal': 35975809990656L, u'total': 35975809990656L, u'free': 21778220282025L}, u'ram': {u'usedByData': 259053897704L, u'used': 803110981632L, u'quotaTotal': 582636011520L, u'total': 809612169216L, u'quotaUsed': 322122547200L, u'quotaUsedPerNode': 107374182400L, u'quotaTotalPerNode': 194212003840L}}, u'diskUsed': 13089975146410L, u'vbActiveNumNonResident': 4859138245L, u'memUsed': 259053897704L} 2021-12-16 14:09:32,609 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | Bucket | Type | Storage Backend | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | ARR | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ | GleamBookUsers0 | couchbase | magma | 1 | none | 0 | 4981627098 | 300.00 GiB | 241.26 GiB | 11.91 TiB | 2.46029610316 | +-----------------+-----------+-----------------+----------+------------+-----+------------+------------+------------+-----------+---------------+ 2021-12-16 14:09:32,609 | test | INFO | MainThread | [table_view:display:72] Docs statistics +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | Initial Items | Current Items | Items Updated | Items Created | Items Deleted | Items Expired | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ | 4000000000 | 38000000000 | 80000000-800000000 | 80000000-800000000 | 40000000-80000000 | 0-0 | +---------------+---------------+--------------------+--------------------+-------------------+---------------+ 2021-12-16 14:09:32,829 | test | INFO | MainThread | [opd:get_bucket_dgm:139] Active Resident Threshold of GleamBookUsers0 is 2.46029610316 2021-12-16 14:09:32,832 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:09:32,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:09:33,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:09:33,213 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:09:33,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:09:33,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:09:33,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:09:33,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:09:33,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:09:33,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:09:33,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:09:33,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 14:09:33,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 14:09:33,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 14:09:34,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 14:09:34,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 14:09:37,772 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-792 is 300 2021-12-16 14:09:37,772 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-720 is 328 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-504 is 313 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-432 is 317 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-216 is 353 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-72 is 324 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.0/kvstore-0 is 349 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-793 is 340 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-721 is 337 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-505 is 353 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-433 is 325 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-361 is 319 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-217 is 359 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-73 is 355 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.1/kvstore-1 is 334 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-802 is 332 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-442 is 357 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-370 is 304 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-226 is 319 2021-12-16 14:09:37,773 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-82 is 351 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.10/kvstore-10 is 335 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-803 is 318 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-443 is 325 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-371 is 322 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-227 is 332 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-83 is 355 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.11/kvstore-11 is 355 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-804 is 355 2021-12-16 14:09:37,775 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-444 is 308 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-228 is 322 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-84 is 346 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.12/kvstore-12 is 310 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-805 is 348 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-445 is 353 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-229 is 323 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-85 is 309 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.13/kvstore-13 is 335 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-806 is 330 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-734 is 325 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-446 is 381 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-374 is 308 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-230 is 345 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.14/kvstore-14 is 335 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-807 is 333 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-735 is 312 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-447 is 383 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-375 is 312 2021-12-16 14:09:37,776 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-231 is 357 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.15/kvstore-15 is 301 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-808 is 323 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-448 is 332 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-376 is 307 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-232 is 324 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.16/kvstore-16 is 349 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-809 is 334 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-449 is 340 2021-12-16 14:09:37,778 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-233 is 348 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.17/kvstore-17 is 334 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-810 is 319 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-450 is 375 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-234 is 337 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.18/kvstore-18 is 344 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-811 is 339 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-739 is 308 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-451 is 332 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-379 is 306 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-235 is 335 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.19/kvstore-19 is 357 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-722 is 323 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-506 is 357 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-434 is 317 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-362 is 313 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-218 is 337 2021-12-16 14:09:37,779 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-74 is 358 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.2/kvstore-2 is 329 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-452 is 346 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-380 is 350 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-236 is 366 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.20/kvstore-20 is 339 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-813 is 317 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-453 is 329 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-381 is 322 2021-12-16 14:09:37,780 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-237 is 334 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.21/kvstore-21 is 356 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-814 is 319 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-742 is 319 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-454 is 331 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-382 is 309 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-238 is 354 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.22/kvstore-22 is 342 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-815 is 329 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-455 is 344 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-383 is 355 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-239 is 307 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.23/kvstore-23 is 336 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-816 is 314 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-456 is 316 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-384 is 305 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-240 is 359 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.24/kvstore-24 is 347 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-817 is 328 2021-12-16 14:09:37,782 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-457 is 346 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-385 is 308 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.25/kvstore-25 is 334 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-818 is 339 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-458 is 344 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-242 is 352 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.26/kvstore-26 is 338 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-819 is 315 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-459 is 316 2021-12-16 14:09:37,783 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-243 is 361 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-171 is 325 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.27/kvstore-27 is 332 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-820 is 327 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-748 is 334 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-244 is 376 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-172 is 312 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.28/kvstore-28 is 304 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-821 is 310 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-749 is 319 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-461 is 319 2021-12-16 14:09:37,785 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-389 is 324 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-245 is 341 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-173 is 346 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.29/kvstore-29 is 334 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-795 is 362 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-723 is 305 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-507 is 353 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-435 is 314 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-219 is 336 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-75 is 322 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.3/kvstore-3 is 331 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-822 is 318 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-750 is 303 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-462 is 340 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-390 is 322 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-246 is 301 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.30/kvstore-30 is 327 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-823 is 314 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-463 is 367 2021-12-16 14:09:37,786 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-391 is 325 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-247 is 347 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-175 is 336 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.31/kvstore-31 is 339 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-824 is 361 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-464 is 364 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-392 is 313 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-248 is 372 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-176 is 338 2021-12-16 14:09:37,788 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.32/kvstore-32 is 332 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-825 is 326 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-753 is 305 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-465 is 308 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-393 is 303 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-249 is 336 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-177 is 318 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.33/kvstore-33 is 377 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-826 is 329 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-466 is 349 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-394 is 309 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-250 is 374 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-178 is 347 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.34/kvstore-34 is 323 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-827 is 309 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-683 is 306 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-467 is 329 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-395 is 309 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-251 is 350 2021-12-16 14:09:37,789 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-179 is 343 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.35/kvstore-35 is 323 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-828 is 339 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-756 is 321 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-468 is 343 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-396 is 313 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-252 is 343 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-180 is 353 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.36/kvstore-36 is 340 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-469 is 336 2021-12-16 14:09:37,790 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-253 is 360 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.37/kvstore-37 is 345 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-830 is 350 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-758 is 334 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-686 is 344 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-470 is 355 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-398 is 325 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-254 is 369 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-182 is 343 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.38/kvstore-38 is 341 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-831 is 323 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-759 is 322 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-471 is 327 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-399 is 330 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-255 is 323 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-183 is 336 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.39/kvstore-39 is 342 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-796 is 302 2021-12-16 14:09:37,792 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-508 is 324 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-436 is 334 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-364 is 302 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-220 is 359 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-76 is 331 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.4/kvstore-4 is 334 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-832 is 343 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-760 is 337 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-688 is 305 2021-12-16 14:09:37,793 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-472 is 362 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-400 is 300 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-184 is 343 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.40/kvstore-40 is 326 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-833 is 326 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-761 is 300 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-473 is 314 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-401 is 311 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-185 is 343 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.41/kvstore-41 is 360 2021-12-16 14:09:37,795 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-834 is 348 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-474 is 331 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-402 is 346 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-186 is 382 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.42/kvstore-42 is 346 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-835 is 325 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-691 is 311 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-475 is 325 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-403 is 319 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-187 is 308 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.43/kvstore-43 is 380 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-836 is 331 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-692 is 326 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-476 is 345 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-404 is 329 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-188 is 363 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.44/kvstore-44 is 346 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-837 is 322 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-477 is 360 2021-12-16 14:09:37,796 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-189 is 349 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.45/kvstore-45 is 350 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-838 is 305 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-478 is 353 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-406 is 319 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-190 is 316 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.46/kvstore-46 is 343 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-839 is 321 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-767 is 329 2021-12-16 14:09:37,798 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-695 is 314 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-407 is 335 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-191 is 361 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.47/kvstore-47 is 332 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-840 is 342 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-696 is 310 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-480 is 317 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-408 is 303 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-192 is 304 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.48/kvstore-48 is 341 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-841 is 341 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-697 is 331 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-481 is 363 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-409 is 318 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-193 is 338 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.49/kvstore-49 is 323 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-797 is 344 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-509 is 350 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-437 is 339 2021-12-16 14:09:37,799 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-221 is 318 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-77 is 352 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.5/kvstore-5 is 336 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-842 is 314 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-770 is 352 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-698 is 335 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-482 is 367 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-410 is 328 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-194 is 353 2021-12-16 14:09:37,801 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.50/kvstore-50 is 362 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-771 is 351 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-699 is 315 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-483 is 352 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-411 is 321 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-195 is 367 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.51/kvstore-51 is 360 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-844 is 361 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-772 is 306 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-700 is 316 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-484 is 334 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-196 is 322 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.52/kvstore-52 is 319 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-845 is 313 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-773 is 334 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-701 is 353 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-485 is 333 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-413 is 316 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-197 is 319 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.53/kvstore-53 is 312 2021-12-16 14:09:37,802 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-846 is 314 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-486 is 348 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-414 is 324 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-342 is 353 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-198 is 363 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.54/kvstore-54 is 332 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-487 is 334 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-343 is 321 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-199 is 336 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.55/kvstore-55 is 341 2021-12-16 14:09:37,803 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-848 is 334 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-776 is 336 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-704 is 331 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-488 is 333 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-416 is 310 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-200 is 358 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.56/kvstore-56 is 345 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-849 is 353 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-705 is 312 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-489 is 342 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-345 is 324 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-201 is 367 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.57/kvstore-57 is 333 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-850 is 337 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-490 is 329 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-418 is 344 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-346 is 321 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-202 is 353 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.58/kvstore-58 is 366 2021-12-16 14:09:37,805 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-851 is 308 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-779 is 314 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-707 is 320 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-491 is 330 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-419 is 309 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-347 is 314 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-203 is 318 2021-12-16 14:09:37,806 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.59/kvstore-59 is 330 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-798 is 345 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-510 is 364 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-438 is 342 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-222 is 319 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-78 is 373 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.6/kvstore-6 is 351 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-852 is 334 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-780 is 337 2021-12-16 14:09:37,808 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-492 is 350 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-204 is 321 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.60/kvstore-60 is 350 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-781 is 350 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-493 is 345 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-421 is 311 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-349 is 324 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.61/kvstore-61 is 341 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-782 is 304 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-710 is 327 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-494 is 321 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-422 is 323 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-350 is 322 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-206 is 342 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.62/kvstore-62 is 335 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-783 is 323 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-495 is 335 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-351 is 339 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-207 is 351 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.63/kvstore-63 is 332 2021-12-16 14:09:37,809 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-352 is 312 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-208 is 324 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.64/kvstore-64 is 321 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-785 is 300 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-497 is 348 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-425 is 312 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-209 is 336 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.65/kvstore-65 is 306 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-786 is 337 2021-12-16 14:09:37,811 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-498 is 330 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-354 is 318 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-210 is 360 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.66/kvstore-66 is 324 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-787 is 325 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-715 is 312 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-499 is 341 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-355 is 313 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-211 is 351 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.67/kvstore-67 is 318 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-788 is 333 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-500 is 342 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-428 is 335 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-356 is 309 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-212 is 326 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.68/kvstore-68 is 351 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-789 is 304 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-501 is 351 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-429 is 352 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-357 is 310 2021-12-16 14:09:37,812 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-213 is 331 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.69/kvstore-69 is 351 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-799 is 333 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-511 is 325 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-439 is 354 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-223 is 336 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-79 is 323 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.7/kvstore-7 is 376 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-790 is 341 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-502 is 343 2021-12-16 14:09:37,813 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-430 is 366 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-358 is 313 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-214 is 363 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.70/kvstore-70 is 348 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-791 is 314 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-503 is 355 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-431 is 331 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-215 is 321 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.71/kvstore-71 is 341 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-800 is 368 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-728 is 304 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-512 is 329 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-440 is 363 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-224 is 308 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-80 is 329 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.8/kvstore-8 is 346 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-801 is 308 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-441 is 333 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-225 is 321 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-81 is 351 2021-12-16 14:09:37,815 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.64--/data/GleamBookUsers0/magma.9/kvstore-9 is 356 2021-12-16 14:09:37,815 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:09:37,819 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:09:37,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:09:38,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:09:38,301 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:09:38,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:09:38,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:09:38,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:09:38,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:09:38,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:09:38,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:09:38,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:09:38,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 14:09:38,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 14:09:38,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 14:09:38,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 14:09:39,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 14:09:41,974 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-1008 is 349 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-864 is 340 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-576 is 310 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-360 is 335 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-144 is 353 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.0/kvstore-72 is 321 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-1009 is 317 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-577 is 338 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-361 is 322 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-145 is 323 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.1/kvstore-73 is 313 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-1018 is 324 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.10/kvstore-514 is 352 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-1019 is 324 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-875 is 333 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-587 is 348 2021-12-16 14:09:41,976 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-515 is 329 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-371 is 349 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-155 is 302 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.11/kvstore-83 is 306 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-948 is 312 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-588 is 351 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.12/kvstore-372 is 330 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-949 is 306 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-877 is 333 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-589 is 301 2021-12-16 14:09:41,977 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-517 is 362 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-373 is 330 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-157 is 344 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.13/kvstore-85 is 353 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-1022 is 313 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-950 is 305 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-518 is 339 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.14/kvstore-374 is 313 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-951 is 351 2021-12-16 14:09:41,979 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-591 is 310 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.15/kvstore-87 is 336 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-952 is 304 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.16/kvstore-88 is 338 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-593 is 344 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-521 is 313 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-377 is 301 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.17/kvstore-89 is 329 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-522 is 332 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-378 is 316 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.18/kvstore-162 is 301 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-955 is 333 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-595 is 332 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-523 is 358 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-163 is 347 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.19/kvstore-91 is 346 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-578 is 350 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-362 is 342 2021-12-16 14:09:41,980 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.2/kvstore-146 is 305 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-596 is 330 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-524 is 300 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.20/kvstore-92 is 334 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-597 is 347 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.21/kvstore-93 is 330 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-526 is 367 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.22/kvstore-382 is 306 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-959 is 309 2021-12-16 14:09:41,982 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-167 is 320 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.23/kvstore-95 is 349 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-960 is 320 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-888 is 327 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-528 is 322 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-384 is 314 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.24/kvstore-168 is 321 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-529 is 319 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.25/kvstore-385 is 341 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-386 is 330 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-170 is 320 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.26/kvstore-98 is 333 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.27/kvstore-387 is 311 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-388 is 305 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.28/kvstore-100 is 341 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-893 is 319 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.29/kvstore-101 is 329 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.3/kvstore-363 is 331 2021-12-16 14:09:41,983 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-966 is 331 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.30/kvstore-534 is 339 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-391 is 360 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.31/kvstore-103 is 336 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-968 is 303 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-536 is 355 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-392 is 304 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.32/kvstore-104 is 322 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-969 is 300 2021-12-16 14:09:41,984 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.33/kvstore-537 is 310 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-970 is 307 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-538 is 340 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.34/kvstore-106 is 309 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-971 is 305 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.35/kvstore-107 is 357 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-540 is 328 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-396 is 341 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.36/kvstore-108 is 310 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.37/kvstore-397 is 314 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.38/kvstore-542 is 334 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-975 is 311 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-543 is 324 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-399 is 319 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.39/kvstore-111 is 304 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-1012 is 326 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.4/kvstore-76 is 301 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-904 is 301 2021-12-16 14:09:41,986 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-544 is 356 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-400 is 309 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.40/kvstore-112 is 325 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.41/kvstore-401 is 309 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.42/kvstore-546 is 326 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-547 is 330 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.43/kvstore-403 is 349 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-548 is 331 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.44/kvstore-404 is 310 2021-12-16 14:09:41,987 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-981 is 303 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-549 is 307 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.45/kvstore-405 is 345 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.46/kvstore-118 is 317 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-983 is 306 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-551 is 339 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.47/kvstore-119 is 356 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.48/kvstore-408 is 353 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-985 is 342 2021-12-16 14:09:41,989 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-409 is 315 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.49/kvstore-121 is 323 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-1013 is 356 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-941 is 349 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-581 is 350 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.5/kvstore-149 is 366 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.50/kvstore-554 is 305 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-915 is 332 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-555 is 337 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.51/kvstore-123 is 329 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-988 is 325 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-556 is 314 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.52/kvstore-412 is 363 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-989 is 307 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-413 is 336 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.53/kvstore-125 is 309 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.54/kvstore-126 is 336 2021-12-16 14:09:41,990 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-343 is 324 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.55/kvstore-127 is 309 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-560 is 309 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.56/kvstore-416 is 324 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-993 is 328 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-561 is 353 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-417 is 320 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.57/kvstore-345 is 314 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-994 is 317 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-418 is 332 2021-12-16 14:09:41,992 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-346 is 320 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.58/kvstore-130 is 329 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-563 is 331 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-419 is 318 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-347 is 339 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.59/kvstore-131 is 325 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-1014 is 323 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-942 is 321 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-582 is 350 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-366 is 334 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.6/kvstore-150 is 304 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-348 is 340 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.60/kvstore-132 is 336 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-565 is 353 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-349 is 326 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.61/kvstore-133 is 324 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-998 is 343 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-926 is 340 2021-12-16 14:09:41,993 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-566 is 337 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-350 is 346 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.62/kvstore-134 is 325 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-567 is 302 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-351 is 338 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.63/kvstore-135 is 346 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.64/kvstore-136 is 303 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-857 is 335 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-353 is 349 2021-12-16 14:09:41,994 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.65/kvstore-137 is 342 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-1002 is 325 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-930 is 301 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-426 is 346 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-354 is 328 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.66/kvstore-138 is 319 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-1003 is 326 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-571 is 318 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-427 is 335 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-355 is 341 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.67/kvstore-139 is 315 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.68/kvstore-356 is 302 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-1005 is 317 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-933 is 314 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-357 is 314 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.69/kvstore-141 is 342 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-1015 is 331 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-943 is 314 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-583 is 306 2021-12-16 14:09:41,996 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.7/kvstore-367 is 356 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-574 is 339 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-358 is 333 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.70/kvstore-142 is 340 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-1007 is 318 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.71/kvstore-575 is 306 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-944 is 342 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-584 is 319 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-368 is 335 2021-12-16 14:09:41,997 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.8/kvstore-152 is 325 2021-12-16 14:09:41,999 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-1017 is 346 2021-12-16 14:09:41,999 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-585 is 341 2021-12-16 14:09:41,999 | test | WARNING | MainThread | [opd:get_magma_disk_usage:635] Number of files in 172.23.110.65--/data/GleamBookUsers0/magma.9/kvstore-81 is 340 2021-12-16 14:09:41,999 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:09:42,002 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:09:42,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:09:42,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:09:42,466 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:09:42,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:09:42,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:09:42,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:09:42,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:09:42,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:09:42,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:09:42,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:09:42,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv* | tail -1 | awk '{print $1}' 2021-12-16 14:09:42,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/wal | tail -1 | awk '{print $1}' 2021-12-16 14:09:42,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-12-16 14:09:43,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: du -cm /data/GleamBookUsers0/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-12-16 14:09:43,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: find /data/GleamBookUsers0/magma*/ -maxdepth 1 -type d -print0 | while read -d "" -r dir; do files=("$dir"/*/*/*); printf "%d,%s " "${#files[@]}" "$dir"; done 2021-12-16 14:09:44,940 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:09:44,941 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:638] Total Disk usage for kvstore is 12539251MB 2021-12-16 14:09:44,941 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:639] Total Disk usage for wal is 6994MB 2021-12-16 14:09:44,943 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:640] Total Disk usage for keyTree is 433617MB 2021-12-16 14:09:44,943 | test | DEBUG | MainThread | [opd:get_magma_disk_usage:641] Total Disk usage for seqTree is 12105271MB 2021-12-16 14:09:44,944 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:09:45,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:09:45,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:09:45,367 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:09:45,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:09:45,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:09:45,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:09:45,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:09:45,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:09:45,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:09:45,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:09:45,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 14:09:45,510 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.65 - core(s): 72 2021-12-16 14:09:45,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 14:09:45,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 14:09:46,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 14:09:47,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 14:09:47,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 14:09:48,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 14:09:48,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 14:09:48,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 14:09:49,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 14:09:49,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 14:09:49,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 14:09:50,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 14:09:50,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 14:09:51,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 14:09:51,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 14:09:51,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 14:09:52,436 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 14:09:52,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 14:09:53,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 14:09:53,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 14:09:53,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 14:09:54,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 14:09:54,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 14:09:55,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 14:09:55,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 14:09:55,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 14:09:56,408 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 14:09:56,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 14:09:57,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 14:09:57,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 14:09:58,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 14:09:58,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 14:09:58,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 14:09:59,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 14:09:59,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 14:10:00,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 14:10:00,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 14:10:00,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 14:10:01,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 14:10:01,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 14:10:02,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 14:10:02,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 14:10:02,778 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 14:10:03,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 14:10:03,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 14:10:03,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 14:10:04,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 14:10:04,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 14:10:05,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 14:10:05,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 14:10:05,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 14:10:06,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 14:10:06,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 14:10:07,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 14:10:07,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 14:10:07,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 14:10:08,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 14:10:08,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 14:10:09,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 14:10:09,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 14:10:09,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 14:10:10,329 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 14:10:10,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 14:10:11,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 14:10:11,460 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:10:11,463 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:10:11,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:10:11,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:10:11,931 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:10:11,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:10:11,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:10:11,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:10:11,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:10:11,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:10:12,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:10:12,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:10:12,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 14:10:12,072 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.66 - core(s): 72 2021-12-16 14:10:12,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 14:10:12,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 14:10:12,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 14:10:13,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 14:10:13,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 14:10:13,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 14:10:14,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 14:10:14,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 14:10:15,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 14:10:15,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 14:10:15,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 14:10:16,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 14:10:16,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 14:10:17,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 14:10:17,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 14:10:17,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 14:10:18,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 14:10:18,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 14:10:19,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 14:10:19,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 14:10:19,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 14:10:20,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 14:10:20,793 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 14:10:21,229 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 14:10:21,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 14:10:22,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 14:10:22,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 14:10:22,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 14:10:23,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 14:10:23,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 14:10:24,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 14:10:24,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 14:10:24,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 14:10:25,155 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 14:10:25,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 14:10:25,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 14:10:26,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 14:10:26,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 14:10:27,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 14:10:27,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 14:10:27,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 14:10:28,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 14:10:28,545 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 14:10:28,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 14:10:29,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 14:10:29,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 14:10:30,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 14:10:30,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 14:10:30,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 14:10:31,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 14:10:31,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 14:10:31,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 14:10:32,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 14:10:32,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 14:10:32,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 14:10:33,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 14:10:33,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 14:10:34,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 14:10:34,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 14:10:34,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 14:10:35,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 14:10:35,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 14:10:36,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 14:10:36,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 14:10:36,740 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:10:36,743 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:10:36,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:10:37,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:10:37,194 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:10:37,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:10:37,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:10:37,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:10:37,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:10:37,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:10:37,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:10:37,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:10:37,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: lscpu | grep 'CPU(s)' | head -1 | awk '{print $2}' 2021-12-16 14:10:37,414 | test | DEBUG | MainThread | [opd:check_fragmentation_using_magma_stats:711] machine: 172.23.110.64 - core(s): 72 2021-12-16 14:10:37,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_0:magma 2021-12-16 14:10:37,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_1:magma 2021-12-16 14:10:38,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_2:magma 2021-12-16 14:10:38,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_3:magma 2021-12-16 14:10:39,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_4:magma 2021-12-16 14:10:39,471 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_5:magma 2021-12-16 14:10:39,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_6:magma 2021-12-16 14:10:40,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_7:magma 2021-12-16 14:10:40,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_8:magma 2021-12-16 14:10:40,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_9:magma 2021-12-16 14:10:41,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_10:magma 2021-12-16 14:10:41,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_11:magma 2021-12-16 14:10:42,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_12:magma 2021-12-16 14:10:42,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_13:magma 2021-12-16 14:10:42,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_14:magma 2021-12-16 14:10:43,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_15:magma 2021-12-16 14:10:43,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_16:magma 2021-12-16 14:10:44,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_17:magma 2021-12-16 14:10:44,456 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_18:magma 2021-12-16 14:10:44,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_19:magma 2021-12-16 14:10:45,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_20:magma 2021-12-16 14:10:45,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_21:magma 2021-12-16 14:10:46,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_22:magma 2021-12-16 14:10:46,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_23:magma 2021-12-16 14:10:46,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_24:magma 2021-12-16 14:10:47,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_25:magma 2021-12-16 14:10:47,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_26:magma 2021-12-16 14:10:47,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_27:magma 2021-12-16 14:10:48,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_28:magma 2021-12-16 14:10:48,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_29:magma 2021-12-16 14:10:49,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_30:magma 2021-12-16 14:10:49,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_31:magma 2021-12-16 14:10:49,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_32:magma 2021-12-16 14:10:50,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_33:magma 2021-12-16 14:10:50,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_34:magma 2021-12-16 14:10:51,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_35:magma 2021-12-16 14:10:51,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_36:magma 2021-12-16 14:10:51,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_37:magma 2021-12-16 14:10:52,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_38:magma 2021-12-16 14:10:52,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_39:magma 2021-12-16 14:10:53,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_40:magma 2021-12-16 14:10:53,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_41:magma 2021-12-16 14:10:53,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_42:magma 2021-12-16 14:10:54,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_43:magma 2021-12-16 14:10:54,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_44:magma 2021-12-16 14:10:54,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_45:magma 2021-12-16 14:10:55,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_46:magma 2021-12-16 14:10:55,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_47:magma 2021-12-16 14:10:55,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_48:magma 2021-12-16 14:10:56,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_49:magma 2021-12-16 14:10:56,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_50:magma 2021-12-16 14:10:57,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_51:magma 2021-12-16 14:10:57,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_52:magma 2021-12-16 14:10:57,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_53:magma 2021-12-16 14:10:58,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_54:magma 2021-12-16 14:10:58,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_55:magma 2021-12-16 14:10:59,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_56:magma 2021-12-16 14:10:59,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_57:magma 2021-12-16 14:10:59,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_58:magma 2021-12-16 14:11:00,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_59:magma 2021-12-16 14:11:00,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_60:magma 2021-12-16 14:11:01,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_61:magma 2021-12-16 14:11:01,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_62:magma 2021-12-16 14:11:01,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 kvstore | grep rw_63:magma 2021-12-16 14:11:02,040 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:11:02,040 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:722] {'172.23.110.65': {'rw_0:magma': {u'NReadBytesGetPerSec': 144244.5625, u'ReplayWALOffset': u'14688:18460672', u'NTablesCreated': 53408, u'RecentReadIOAmpSet': 0.03587478771805763, u'ReadIOAmp': 1.2077662944793701, u'BlockCacheMisses': 2906253, u'NFlushes': 14936, u'RecentWriteAmp': 2.511390447616577, u'NumSyncFlushes': 901, u'TotalMemUsed': 135043582, u'NBloomFilterHits': 30045754, u'Fragmentation': 0.2822031080722809, u'NumNormalFlushes': 8063, u'RecentBlockCacheHitRatio': 36.32838821411133, u'seqStats': {u'NItemsTransformed': 0, u'NTablesCreated': 22815, u'ReadIOAmp': 0.0, u'MemtablesMemUsed': 3629264, u'NBloomFilterHits': 0, u'TotalIndexBlocksSize': 493767324, u'CheckpointOverhead': 929792, u'NItemsGet': 0, u'NWriteIOs': 49583, u'NReadBytes': 16831153, u'LevelStats': [{u'TargetSize': 41943040, u'LogicalSize': 273266, u'SizeRatio': 0.0065151691436767575, u'NWriteBytes': 69410816, u'BloomFilterMemUsed': 0, u'NumTables': 77, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 630784}, {u'TargetSize': 41943040, u'LogicalSize': 33060065, u'SizeRatio': 0.7882133722305298, u'NWriteBytes': 21184512, u'BloomFilterMemUsed': 0, u'NumTables': 20, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 462848}, {u'TargetSize': 661135654, u'LogicalSize': 462203196, u'SizeRatio': 0.699104931345905, u'NWriteBytes': 315392, u'BloomFilterMemUsed': 0, u'NumTables': 160, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 27258880}, {u'TargetSize': 33816301017L, u'LogicalSize': 19465579563L, u'SizeRatio': 0.5756271081575226, u'NWriteBytes': 36864, u'BloomFilterMemUsed': 0, u'NumTables': 913, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 190263296}, {u'TargetSize': 132227132078L, u'LogicalSize': 67632602040L, u'SizeRatio': 0.5114880809795067, u'NWriteBytes': 12860354560L, u'BloomFilterMemUsed': 0, u'NumTables': 491, u'WastedSpace': 1519036001, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 67632496640L}], u'NItemsGarbageCollected': 1325960, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 414113942, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 175008, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 46088, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 514086, u'CompactWriter': {u'NDataBlockWrites': 6222, u'NWriteBytes': 21184512, u'NIndexBlockWrites': 937, u'NBytesIncoming': 22450142, u'NSyncs': 0, u'NInserts': 1020461, u'NWriteIOs': 937}, u'NWriteBytes': 12990252757L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'NBloomFilterMisses': 0, u'NItemsMerged': 0, u'LogicalDiskSize': 67632602040L, u'TotalBloomFilterMemUsed': 0, u'WriteAmp': 0.6961759753957432, u'BufferMemUsed': 456618, u'ActiveBloomFilterMemUsed': 0, u'LogicalDataSize': 48167022477L, u'NTableFiles': 1773, u'NFileCountCompacts': 498, u'TableObjectMemUsed': 452104, u'TreeSnapshotMemoryUsed': 206112, u'TotalDiskUsage': 81365012480L, u'BytesIncoming': 18659438441L, u'NItemsDel': 0, u'PendingCheckpointBytes': 212992, u'NCompacts': 510, u'NItemsPut': 14320450, u'NSyncs': 6873, u'CompactReader': {u'NReadIOs': 23950, u'NReadBytes': 15381273, u'NDataBlockReads': 14562, u'NBytesOutgoing': 0, u'NIndexBlockReads': 9388}, u'ActiveDiskUsage': 67851112448L, u'NTablesDeleted': 9388, u'IterateReader': {u'NReadIOs': 22138, u'NReadBytes': 1449880, u'NDataBlockReads': 11069, u'NBytesOutgoing': 0, u'NIndexBlockReads': 11069}, u'BloomFilterFPR': 0.0, u'Writer': {u'NDataBlockWrites': 3894607, u'NWriteBytes': 12969068245L, u'NIndexBlockWrites': 74218, u'NBytesIncoming': 16827080428L, u'NSyncs': 6873, u'NInserts': 11665345, u'NWriteIOs': 48646}}, u'NReadBytes': 11132805593L, u'HistogramMemUsed': 101776, u'BlockCacheMissesPerSec': 251.8000030517578, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 441458480, u'NActivePurgers': 0, u'walStats': {u'TxnSizeEstimate': 13201, u'NTxns': 985725, u'NWriteBytes': 22434074624L, u'NSegments': 565, u'NSyncs': 985724, u'NLoggedBytes': 22434074624L, u'NActiveSegments': 1, u'NWrites': 1045745, u'NInserts': 18929655, u'DiskUsed': 24293376}, u'NGets': 1191820, u'NReadBytesSet': 329035969, u'NReadBytesCompact': 6849704940L, u'IndexResidentRatio': 0.03768337517976761, u'CompactQueueSize': 0, u'NSyncsPerSec': 3.0, u'BlockCacheHitRatio': 32.18292999267578, u'ReadAheadBufferMemUsed': 0, u'NReadBytesGet': 4281650773L, u'BloomFilterMemUsedRatio': 0.29763263463974, u'NWriteBytes': 43115035168L, u'BytesOutgoing': 0, u'RecentReadIOAmp': 0.521270215511322, u'WriteAmp': 2.1627769470214844, u'NBlocksCached': 24355, u'LogicalDataSize': 49862714342L, u'ActiveBloomFilterMemUsed': 44305448, u'NFileCountCompacts': 2069, u'NTableFiles': 2561, u'FlushQueueSize': 72, u'TableObjectMemUsed': 647528, u'BlockCacheQuota': 82779041, u'TotalDiskUsage': 83406557184L, u'NPurgerRuns': 19, u'ReadIOAmpSet': 0.0, u'NSyncs': 20619, u'NReadIOsSetPerSec': 15.433333396911621, u'NReadIOsPerSec': 286.3666687011719, u'NTablesDeleted': 40050, u'BloomFilterFPR': 0.020943591371178627, u'NBloomFilterHitsPerSec': 2828.7333984375, u'WriteCacheMemUsed': 5235312, u'RecentBloomFilterFPR': 0.023261167109012604, u'BytesPerRead': 3316.452880859375, u'BytesIncomingPerSec': 847772.0625, u'NReadIOsGetPerSec': 58.0, u'NGetStats': 9706, u'TotalIndexBlocksSize': 521458291, u'NGetStatsPerSec': 0.9666666388511658, u'NWriteBytesCompact': 6733615104L, u'NReadIOsSet': 127355, u'ActiveDataSize': 49862714342L, u'NWriteBytesCompactPerSec': 585728.0, u'NBloomFilterMissesPerSec': 67.36666870117188, u'NCommitBatches': 985724, u'NGetStatsComputed': 9706, u'NWriteIOs': 1152611, u'NReadIOsGet': 1566795, u'RecentReadAmpGet': 0.0, u'WriteCacheQuota': 20694760, u'LSMTreeObjectMemUsed': 165760, u'NSetsPerSec': 430.20001220703125, u'NDataLevelFileCountCompacts': 0, u'NFlushesPerSec': 2.0, u'ManagedFileObjectMemUsed': 250656, u'ReadAmpGet': 0.0, u'BlockCacheMemUsed': 82501696, u'NWriterCompacts': 0, u'NReadIOs': 4158780, u'WALBufferMemUsed': 65536, u'ReadAmp': 0.0, u'keyStats': {u'NItemsTransformed': 159915563, u'NTablesCreated': 16395, u'ReadIOAmp': 0.327106901600984, u'MemtablesMemUsed': 303312, u'NBloomFilterHits': 30043802, u'TotalIndexBlocksSize': 27680287, u'CheckpointOverhead': 23916544, u'NItemsGet': 4789367, u'NWriteIOs': 36212, u'NReadBytes': 11089588203L, u'LevelStats': [{u'TargetSize': 41943040, u'LogicalSize': 1172746, u'SizeRatio': 0.027960443496704103, u'NWriteBytes': 852742144, u'BloomFilterMemUsed': 11472, u'NumTables': 71, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 1073152}, {u'TargetSize': 41943040, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 41943040, u'LogicalSize': 33375669, u'SizeRatio': 0.7957379579544067, u'NWriteBytes': 2968371200L, u'BloomFilterMemUsed': 1047552, u'NumTables': 20, u'WastedSpace': 83487, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 33456128}, {u'TargetSize': 183360898, u'LogicalSize': 138036834, u'SizeRatio': 0.7528149976665145, u'NWriteBytes': 1252044800, u'BloomFilterMemUsed': 4612736, u'NumTables': 72, u'WastedSpace': 381571, u'CompactingSize': 0, u'Level': 3, u'PhysicalSize': 138407936}, {u'TargetSize': 1833609021, u'LogicalSize': 1833609021, u'SizeRatio': 1.0, u'NWriteBytes': 2599493632L, u'BloomFilterMemUsed': 38632448, u'NumTables': 255, u'WastedSpace': 6465369, u'CompactingSize': 0, u'Level': 4, u'PhysicalSize': 1840070656}], u'NItemsGarbageCollected': 1715007, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 27341496, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 54720, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 4084096, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 173230, u'CompactWriter': {u'NDataBlockWrites': 2456207, u'NWriteBytes': 6705958912L, u'NIndexBlockWrites': 31423, u'NBytesIncoming': 9380873350L, u'NSyncs': 0, u'NInserts': 163693320, u'NWriteIOs': 14923}, u'NWriteBytes': 7570809971L, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 1566635, u'NReadBytes': 4281493629L, u'NDataBlockReads': 3151204, u'NBytesOutgoing': 0, u'NIndexBlockReads': 4906924}, u'NBloomFilterMisses': 642615, u'NItemsMerged': 0, u'LogicalDiskSize': 1833609021, u'TotalBloomFilterMemUsed': 44384952, u'WriteAmp': 11.909730588981258, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 44304208, u'LogicalDataSize': 1695572187, u'NTableFiles': 570, u'NFileCountCompacts': 781, u'TableObjectMemUsed': 141360, u'TreeSnapshotMemoryUsed': 88864, u'TotalDiskUsage': 2039758848, u'BytesIncoming': 635682723, u'NItemsDel': 0, u'PendingCheckpointBytes': 5500928, u'NCompacts': 1227, u'NItemsPut': 12973127, u'NSyncs': 6873, u'CompactReader': {u'NReadIOs': 2517461, u'NReadBytes': 6808094574L, u'NDataBlockReads': 2472847, u'NBytesOutgoing': 0, u'NIndexBlockReads': 44614}, u'ActiveDiskUsage': 2013007872, u'NTablesDeleted': 16444, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.02094135004422315, u'Writer': {u'NDataBlockWrites': 188023, u'NWriteBytes': 864851059, u'NIndexBlockWrites': 15095, u'NBytesIncoming': 675409210, u'NSyncs': 6873, u'NInserts': 11644995, u'NWriteIOs': 21289}}, u'NReadBytesCompactPerSec': 580209.6875, u'TableMetaMemUsed': 749892, u'NDeletes': 0, u'BlockCacheHitsPerSec': 143.6666717529297, u'BytesOutgoingPerSec': 0.0, u'NBloomFilterMisses': 642727, u'LogicalDiskSize': 69466330739L, u'NSets': 3597547, u'TotalBloomFilterMemUsed': 44386192, u'BufferMemUsed': 456618, u'RecentReadAmp': 0.0, u'NGetsPerSec': 111.26667022705078, u'NInserts': 9375580, u'NCommitBatchesPerSec': 157.86666870117188, u'FragmentationWithOverhead': 0.4021727740764618, u'NCompactsPerSec': 0.4333333373069763, u'WALMemUsed': 131152, u'MemoryQuota': 149130800, u'NReadBytesPerSec': 724454.25, u'TreeSnapshotMemoryUsed': 335072, u'BytesIncoming': 19935035388L, u'NCompacts': 2527, u'NumPersistentFlushes': 5972, u'NWriteIOsPerSec': 170.1666717529297, u'NInsertsPerSec': 0.0, u'RecentBytesPerRead': 1296.3861083984375, u'ActiveDiskUsage': 69864620032L, u'NDeletesPerSec': 0.0, u'NGetStatsComputedPerSec': 0.9666666388511658, u'NReadBytesSetPerSec': 38841.265625, u'BlockCacheHits': 1379177, u'NWriteBytesPerSec': 2129086.75, u'localStats': {u'NItemsTransformed': 0, u'NTablesCreated': 14198, u'ReadIOAmp': 0.054533060668029994, u'MemtablesMemUsed': 1302736, u'NBloomFilterHits': 1952, u'TotalIndexBlocksSize': 10680, u'CheckpointOverhead': 1286144, u'NItemsGet': 2934, u'NWriteIOs': 21071, u'NReadBytes': 26386237, u'LevelStats': [{u'TargetSize': 41943040, u'LogicalSize': 517171, u'SizeRatio': 0.012330317497253418, u'NWriteBytes': 109846528, u'BloomFilterMemUsed': 600, u'NumTables': 51, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 0, u'PhysicalSize': 417792}, {u'TargetSize': 41943040, u'LogicalSize': 0, u'SizeRatio': 0.0, u'NWriteBytes': 0, u'BloomFilterMemUsed': 0, u'NumTables': 0, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 1, u'PhysicalSize': 0}, {u'TargetSize': 41943040, u'LogicalSize': 119678, u'SizeRatio': 0.0028533458709716795, u'NWriteBytes': 6471680, u'BloomFilterMemUsed': 640, u'NumTables': 10, u'WastedSpace': 0, u'CompactingSize': 0, u'Level': 2, u'PhysicalSize': 81920}], u'NItemsGarbageCollected': 2997465, u'NInternalFragmentationCompacts': 0, u'NTTLCompacts': 0, u'ActiveIndexBlocksSize': 3042, u'NDataLevelFileCountCompacts': 0, u'ManagedFileObjectMemUsed': 20928, u'NNonL0LocalCompacts': 0, u'NWriterCompacts': 0, u'NReadIOs': 28596, u'UID': u'', u'ReadAmp': 0.0, u'NItemsDropped': 0, u'NFailedCompacts': 0, u'ReadAheadBufferMemUsed': 0, u'TableMetaMemUsed': 62576, u'CompactWriter': {u'NDataBlockWrites': 790, u'NWriteBytes': 6471680, u'NIndexBlockWrites': 790, u'NBytesIncoming': 4362380, u'NSyncs': 0, u'NInserts': 41870, u'NWriteIOs': 790}, u'NWriteBytes': 119897816, u'BytesOutgoing': 0, u'Reader': {u'NReadIOs': 160, u'NReadBytes': 157144, u'NDataBlockReads': 561, u'NBytesOutgoing': 0, u'NIndexBlockReads': 561}, u'NBloomFilterMisses': 112, u'NItemsMerged': 0, u'LogicalDiskSize': 119678, u'TotalBloomFilterMemUsed': 1240, u'WriteAmp': 0.08275722727923647, u'BufferMemUsed': 0, u'ActiveBloomFilterMemUsed': 1240, u'LogicalDataSize': 119678, u'NTableFiles': 218, u'NFileCountCompacts': 790, u'TableObjectMemUsed': 54064, u'TreeSnapshotMemoryUsed': 40096, u'TotalDiskUsage': 1785856, u'BytesIncoming': 1448789670, u'NItemsDel': 0, u'PendingCheckpointBytes': 0, u'NCompacts': 790, u'NItemsPut': 2999376, u'NSyncs': 6873, u'CompactReader': {u'NReadIOs': 28436, u'NReadBytes': 26229093, u'NDataBlockReads': 14218, u'NBytesOutgoing': 0, u'NIndexBlockReads': 14218}, u'ActiveDiskUsage': 499712, u'NTablesDeleted': 14218, u'IterateReader': {u'NReadIOs': 0, u'NReadBytes': 0, u'NDataBlockReads': 0, u'NBytesOutgoing': 0, u'NIndexBlockReads': 0}, u'BloomFilterFPR': 0.05426356589147287, u'Writer': {u'NDataBlockWrites': 13408, u'NWriteBytes': 113426136, u'NIndexBlockWrites': 13408, u'NBytesIncoming': 21796978, u'NSyncs': 6873, u'NInserts': 107697, u'NWriteIOs': 20281}}}}} 2021-12-16 14:11:02,045 | test | INFO | MainThread | [opd:check_fragmentation_using_magma_stats:727] magma stats fragmentation result {'172.23.110.65': [0.2822031080722809, 0.21997296810150146, 0.2116669863462448, 0.2112693041563034, 0.24171005189418793, 0.25678467750549316, 0.2581949532032013, 0.22020553052425385, 0.25600722432136536, 0.2617220878601074, 0.2611534893512726, 0.29253464937210083, 0.29833516478538513, 0.2930435836315155, 0.26021668314933777, 0.24148952960968018, 0.24950885772705078, 0.25541919469833374, 0.27355456352233887, 0.2596709430217743, 0.23984307050704956, 0.2532431483268738, 0.2458878755569458, 0.25610148906707764, 0.24727053940296173, 0.25137946009635925, 0.21630357205867767, 0.16909559071063995, 0.17953284084796906, 0.21611079573631287, 0.20699270069599152, 0.21841968595981598, 0.1694241315126419, 0.177537202835083, 0.17726287245750427, 0.2300003468990326, 0.1968538463115692, 0.20333272218704224, 0.19300885498523712, 0.20491252839565277, 0.23021887242794037, 0.2251037061214447, 0.25019365549087524, 0.227945938706398, 0.22045163810253143, 0.20979969203472137, 0.17933443188667297, 0.21594463288784027, 0.2172422707080841, 0.21837827563285828, 0.21941854059696198, 0.26487576961517334, 0.2087283581495285, 0.22078678011894226, 0.23655648529529572, 0.22894695401191711, 0.27797025442123413, 0.23051781952381134, 0.22915750741958618, 0.2741457521915436, 0.24717961251735687, 0.23625409603118896, 0.2496403604745865, 0.18834155797958374], '172.23.110.66': [0.0029293743427842855, 0.00264229578897357, 0.002730358624830842, 0.0028117273468524218, 0.002733930479735136, 0.0025064232759177685, 0.0028265193104743958, 0.002801015740260482, 0.0028394272085279226, 0.002916688099503517, 0.0030358980875462294, 0.0026088394224643707, 0.0029460436198860407, 0.0028940190095454454, 0.002680838806554675, 0.002958552911877632, 0.0028919572941958904, 0.0027193520218133926, 0.0025595021434128284, 0.0024764249101281166, 0.002863801084458828, 0.0027816907968372107, 0.0028494002763181925, 0.0027463207952678204, 0.0025343685410916805, 0.002812132937833667, 0.002873960416764021, 0.0028345216996967793, 0.002860786160454154, 0.002681248588487506, 0.002774380147457123, 0.0027462674770504236, 0.00262458436191082, 0.0025764964520931244, 0.0025584648828953505, 0.0028502780478447676, 0.0028780789580196142, 0.002654234878718853, 0.0027645218651741743, 0.002753519918769598, 0.0029236217960715294, 0.0028299137484282255, 0.0028874690178781748, 0.002820051973685622, 0.0027801506221294403, 0.0029537316877394915, 0.002799229696393013, 0.0027191564440727234, 0.002761350478976965, 0.002736110473051667, 0.0027034662198275328, 0.0028982097283005714, 0.0027731661684811115, 0.0027295241598039865, 0.0026884074322879314, 0.002939525991678238, 0.0028957652393728495, 0.0027927597984671593, 0.002816066611558199, 0.002794726984575391, 0.002854678314179182, 0.0028946297243237495, 0.0029634961392730474, 0.0025610988959670067], '172.23.110.64': [0.2838328182697296, 0.30886974930763245, 0.31339070200920105, 0.29869869351387024, 0.30065882205963135, 0.2528155744075775, 0.2816106975078583, 0.24417704343795776, 0.2699671983718872, 0.23707352578639984, 0.2439018189907074, 0.25021225214004517, 0.25068986415863037, 0.224745973944664, 0.2693282663822174, 0.25873419642448425, 0.22523939609527588, 0.2323242723941803, 0.19870610535144806, 0.2726360857486725, 0.23059751093387604, 0.24268075823783875, 0.2520621120929718, 0.24999774992465973, 0.26455551385879517, 0.25355830788612366, 0.20511935651302338, 0.25980696082115173, 0.26456665992736816, 0.2968152165412903, 0.2997792661190033, 0.2603614926338196, 0.27545398473739624, 0.3023667335510254, 0.2567637264728546, 0.2794029712677002, 0.28409498929977417, 0.2592228949069977, 0.3161401152610779, 0.2799232304096222, 0.30468714237213135, 0.24016746878623962, 0.22288288176059723, 0.2465532422065735, 0.2527253031730652, 0.19293442368507385, 0.21289408206939697, 0.27674877643585205, 0.24656380712985992, 0.2605026960372925, 0.2838852107524872, 0.2788090705871582, 0.2685962915420532, 0.2718667685985565, 0.30351927876472473, 0.1891213357448578, 0.32247963547706604, 0.25561946630477905, 0.2892405390739441, 0.30842095613479614, 0.22350090742111206, 0.27466726303100586, 0.2733880877494812, 0.26730695366859436]} 2021-12-16 14:11:02,309 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.65 is 18.0 2021-12-16 14:11:02,618 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.66 is 18.0 2021-12-16 14:11:02,825 | test | DEBUG | MainThread | [opd:check_fragmentation_using_kv_stats:672] Current Fragmentation for node 172.23.110.64 is 18.0 2021-12-16 14:11:02,825 | test | INFO | MainThread | [opd:check_fragmentation_using_kv_stats:675] KV stats fragmentation values {'172.23.110.65': 18.0, '172.23.110.66': 18.0, '172.23.110.64': 18.0} ############################################################ # # RAM FootPrint: 4033.10546875 # ############################################################ 2021-12-16 14:11:02,877 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:11:02,878 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 98 seconds. Reason: Iteration:0 waiting to kill memc on all nodes 2021-12-16 14:12:40,881 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:12:40,882 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:12:45,884 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:12:46,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:46,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:12:46,325 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:46,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:12:46,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:12:46,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:12:46,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:12:46,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:12:46,410 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:12:46,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:12:46,440 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:12:46,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:12:46,507 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['286379 memcached\n'], []) 2021-12-16 14:12:46,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:12:46,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:12:46,615 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['286379 memcached\n'], []) 2021-12-16 14:12:46,615 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:12:46,618 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:12:46,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:46,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:12:47,039 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:47,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:12:47,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:12:47,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:12:47,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:12:47,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:12:47,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:12:47,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:12:47,144 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:12:47,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:12:47,190 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['184119 memcached\n'], []) 2021-12-16 14:12:47,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:12:47,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:12:47,332 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['184119 memcached\n'], []) 2021-12-16 14:12:47,334 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:12:47,336 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:12:47,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:47,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:12:47,813 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:47,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:12:47,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:12:47,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:12:47,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:12:47,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:12:47,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:12:47,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:12:47,987 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:12:47,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:12:48,078 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['22316 memcached\n'], []) 2021-12-16 14:12:48,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:12:48,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:12:48,213 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['22316 memcached\n'], []) 2021-12-16 14:12:48,213 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:12:48,216 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:12:48,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:48,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:12:48,658 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:48,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:12:48,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:12:48,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:12:48,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:12:48,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:12:48,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:12:48,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:12:48,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:12:48,803 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:12:48,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:12:48,854 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:48,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:48,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:48,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:48,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:49,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:49,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:49,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:49,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:49,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:49,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:49,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:49,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:49,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:49,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:49,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:49,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:49,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:49,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:50,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:50,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:50,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:50,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:50,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:50,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,410 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:50,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:50,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:50,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:50,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:50,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:50,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:50,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:50,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:50,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:50,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:51,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:51,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:12:51,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:12:51,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:12:51,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:12:51,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:12:51,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:12:51,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:12:51,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:12:51,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:51,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:12:51,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:12:51,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:12:51,697 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:12:51,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:12:51,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:51,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:12:51,744 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:12:51,744 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:12:51,747 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:12:51,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:52,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:12:52,255 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:52,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:12:52,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:12:52,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:12:52,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:12:52,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:12:52,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:12:52,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:12:52,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:12:52,411 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:12:52,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:12:52,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:52,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:52,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:12:52,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:12:52,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:52,655 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:52,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:52,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:52,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:52,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:52,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:52,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:52,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:52,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:52,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:53,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:53,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:53,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:53,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:53,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:53,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:53,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:53,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:53,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:53,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,655 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:53,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:53,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:53,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:53,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:53,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:53,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:53,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:53,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:54,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:54,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:54,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:54,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:54,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:54,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:54,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:54,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,589 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:54,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:54,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:12:54,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:12:54,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:12:54,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:12:54,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:54,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:54,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:12:55,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:12:55,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:55,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:55,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:12:55,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:12:55,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:12:55,221 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:12:55,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:12:55,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:55,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:12:55,285 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:12:55,285 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:12:55,288 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:12:55,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:55,727 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:12:55,805 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:55,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:12:55,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:12:55,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:12:55,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:12:55,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:12:55,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:12:55,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:12:55,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:12:55,927 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:12:55,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:12:55,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:12:56,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:12:56,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:12:56,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:12:56,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:12:56,433 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:12:56,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:12:56,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:12:56,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:12:56,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:12:56,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:56,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:12:56,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:56,943 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:56,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:57,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:12:57,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:57,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:12:57,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:57,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:12:57,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:57,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:12:57,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:57,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:12:57,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:57,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:12:57,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:57,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:12:57,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:57,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:57,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:57,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:12:57,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:58,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:12:58,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:58,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:12:58,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:58,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:12:58,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:58,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:12:58,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:58,553 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:12:58,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:12:58,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:58,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:12:58,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:12:58,661 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:12:58,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:12:58,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:12:58,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:12:58,697 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:12:58,698 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:12:58,698 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:12:58,700 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:12:58,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:12:59,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:12:59,187 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:12:59,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:12:59,204 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:12:59,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:12:59,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:12:59,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:12:59,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:12:59,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:12:59,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:01,604 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-16 14:13:01,608 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:01,608 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:03,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:03,790 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:03,798 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:03,799 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:05,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:05,980 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:05,984 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:05,984 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:07,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:08,091 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:08,095 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:08,095 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:10,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:10,256 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:10,263 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:10,263 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:12,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:12,431 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:12,436 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:12,436 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:14,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:14,598 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:14,601 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:14,601 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:16,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:16,775 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:16,779 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:16,779 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:18,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:18,948 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:18,953 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:18,953 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:20,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:21,112 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:21,117 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:21,117 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:23,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:23,286 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:23,290 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:23,290 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:25,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:25,461 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:25,467 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:25,467 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:27,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:27,628 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:27,635 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:27,635 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:29,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:29,819 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:29,825 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:29,825 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:31,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:32,000 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:13:32,005 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:32,006 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:34,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:34,155 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:34,155 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:36,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:36,312 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:36,312 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:38,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:38,487 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:38,489 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:40,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:40,680 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:40,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:42,681 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:42,848 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:42,849 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:44,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:45,003 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:45,005 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:47,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:47,163 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:47,164 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:49,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:49,305 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:49,306 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:51,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:51,454 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:51,454 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:53,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:53,624 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:53,624 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:55,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:55,779 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:55,780 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:57,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:13:57,940 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:13:57,941 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:13:59,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:00,107 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:00,107 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:02,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:02,282 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:02,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:04,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:04,440 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:04,441 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:06,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:06,607 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:06,608 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:08,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:08,766 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:08,766 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:10,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:10,923 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:10,923 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:12,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:13,089 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:13,089 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:15,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:15,240 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:15,242 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:17,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:17,401 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:17,401 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:19,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:19,555 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:19,555 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:21,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:21,723 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:21,723 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:23,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:23,884 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:23,885 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:25,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:26,049 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:26,049 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:28,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:28,227 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:28,227 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:30,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:30,405 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:30,405 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:32,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:32,575 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:32,575 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:34,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:34,743 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:34,744 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:36,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:36,924 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:36,924 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:38,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:39,101 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:39,101 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:41,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:41,280 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:41,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:43,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:43,466 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:43,466 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:14:45,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:45,625 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:14:45,625 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 106.926000118. 2021-12-16 14:14:45,625 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:14:45,628 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:14:45,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:14:46,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:14:46,174 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:14:46,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:14:46,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:14:46,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:14:46,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:14:46,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:14:46,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:14:46,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:14:46,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:46,490 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:14:46,490 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.865999937057. 2021-12-16 14:14:46,490 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:14:46,493 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:14:46,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:14:46,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:14:46,901 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:14:46,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:14:46,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:14:46,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:14:46,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:14:47,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:14:47,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:14:47,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:14:47,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:47,246 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:47,246 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:14:49,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:49,436 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:49,437 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:14:51,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:51,658 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:51,658 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:14:53,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:53,831 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:53,832 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:14:55,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:56,000 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:56,000 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:14:58,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:14:58,184 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:14:58,184 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:15:00,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:15:00,358 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:15:00,358 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:15:02,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:15:02,542 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:15:02,542 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:15:04,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:15:04,835 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:15:04,836 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 18.3459999561. ############################################################ # # RAM FootPrint: 3513.328125 # ############################################################ 2021-12-16 14:15:04,871 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:15:04,871 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:1 waiting to kill memc on all nodes 2021-12-16 14:16:12,875 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:16:12,875 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:16:17,877 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:16:18,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:19,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:16:19,292 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:19,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:16:19,346 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:16:19,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:16:19,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:16:19,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:16:19,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:16:19,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:16:19,509 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:16:19,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:16:19,599 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['301437 memcached\n'], []) 2021-12-16 14:16:19,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:16:19,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:16:19,795 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['301437 memcached\n'], []) 2021-12-16 14:16:19,795 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:16:19,798 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:16:19,908 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:20,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:16:20,322 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:20,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:16:20,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:16:20,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:16:20,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:16:20,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:16:20,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:16:20,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:16:20,444 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:16:20,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:16:20,513 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['198694 memcached\n'], []) 2021-12-16 14:16:20,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:16:20,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:16:20,640 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['198694 memcached\n'], []) 2021-12-16 14:16:20,640 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:16:20,642 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:16:20,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:21,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:16:21,313 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:21,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:16:21,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:16:21,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:16:21,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:16:21,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:16:21,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:16:21,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:16:21,456 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:16:21,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:16:21,522 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['37833 memcached\n'], []) 2021-12-16 14:16:21,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:16:21,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:16:21,678 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['37833 memcached\n'], []) 2021-12-16 14:16:21,678 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:16:21,681 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:16:21,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:22,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:16:22,142 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:22,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:16:22,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:16:22,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:16:22,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:16:22,224 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:16:22,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:16:22,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:16:22,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:16:22,290 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:16:22,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:16:22,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:22,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:22,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:22,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:22,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:22,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:22,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:22,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:22,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:22,683 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:22,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:22,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:22,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:22,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:22,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:22,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:22,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:22,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:22,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:23,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:23,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:23,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:23,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:23,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:23,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:23,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:23,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:23,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:23,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:23,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:23,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:23,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:23,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:23,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:23,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:23,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:23,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:24,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:24,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:24,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:24,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,316 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:24,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:24,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:16:24,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:16:24,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:16:24,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:16:24,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:16:24,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:16:24,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:24,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:16:24,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:16:24,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:24,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:25,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:16:25,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:16:25,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:16:25,066 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:16:25,066 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:16:25,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:25,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:16:25,109 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:16:25,111 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:16:25,112 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:16:25,219 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:25,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:16:25,611 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:25,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:16:25,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:16:25,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:16:25,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:16:25,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:16:25,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:16:25,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:16:25,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:16:25,763 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:16:25,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:16:25,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:25,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:25,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:25,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:25,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:26,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:26,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:26,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:26,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:26,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:26,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:26,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:26,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:26,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:26,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:26,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:26,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:26,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:26,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:26,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:26,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:26,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:26,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:26,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:27,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:27,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:27,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:27,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:27,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:27,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:27,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:27,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:27,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:27,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:27,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:16:27,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:16:27,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:16:27,655 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:16:27,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:16:27,727 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:16:27,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:16:27,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:16:27,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:27,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:16:27,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:16:27,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:16:27,920 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:16:27,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:16:27,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:27,946 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:16:27,960 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:16:27,960 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:16:27,963 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:16:28,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:28,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:16:28,486 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:28,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:16:28,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:16:28,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:16:28,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:16:28,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:16:28,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:16:28,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:16:28,617 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:16:28,635 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:16:28,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:16:28,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:28,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:28,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:16:28,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:16:28,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:28,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:28,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:16:28,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:16:28,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,066 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:16:29,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:16:29,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:16:29,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:16:29,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,267 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:16:29,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:16:29,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:29,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:16:29,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:29,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:16:29,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:29,571 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:16:29,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:29,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:16:29,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:29,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:16:29,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:29,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:16:29,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:29,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:16:29,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:29,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:29,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:16:29,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:29,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:30,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:16:30,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:30,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:16:30,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:30,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:16:30,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:30,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:16:30,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:30,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:16:30,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:30,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:16:30,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:16:30,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:30,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:16:30,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:16:30,492 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:16:30,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:16:30,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:16:30,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:16:30,542 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:16:30,542 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:16:30,542 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:16:30,546 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:16:30,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:16:31,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:16:31,138 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:16:31,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:16:31,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:16:31,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:16:31,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:16:31,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:16:31,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:16:31,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:16:31,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:31,400 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:31,405 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:31,405 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:33,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:33,578 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:33,582 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:33,582 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:35,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:35,767 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:35,772 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:35,772 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:37,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:37,941 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:37,946 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:37,946 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:39,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:40,125 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:40,130 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:40,130 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:42,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:42,312 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:42,316 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:42,316 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:44,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:44,493 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:44,496 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:44,496 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:46,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:46,674 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:46,678 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:46,678 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:48,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:48,858 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:48,862 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:48,864 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:50,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:51,026 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:51,030 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:51,030 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:53,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:53,188 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:53,194 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:53,194 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:55,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:55,365 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:16:55,368 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:55,368 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:57,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:57,516 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:57,517 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:16:59,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:16:59,664 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:16:59,664 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:01,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:01,812 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:01,812 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:03,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:03,964 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:03,964 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:05,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:06,118 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:06,118 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:08,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:08,273 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:08,273 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:10,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:10,421 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:10,421 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:12,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:12,614 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:12,614 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:14,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:14,776 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:14,776 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:16,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:16,936 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:16,937 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:18,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:19,092 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:19,092 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:21,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:21,249 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:21,250 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:23,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:23,424 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:23,424 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:25,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:25,596 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:25,596 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:27,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:27,759 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:27,759 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:29,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:29,921 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:29,921 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:31,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:32,086 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:32,086 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:34,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:34,252 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:34,253 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:36,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:36,404 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:36,404 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:38,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:38,559 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:38,561 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:40,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:40,717 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:40,717 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:42,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:43,075 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:43,075 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:45,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:45,259 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:45,260 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:47,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:47,437 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:47,437 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:49,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:49,792 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:49,792 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:51,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:51,956 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:51,957 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:53,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:54,115 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:54,117 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:56,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:56,282 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:56,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:17:58,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:17:58,456 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:17:58,456 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:18:00,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:00,677 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:00,677 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:18:02,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:03,164 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:03,164 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:18:05,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:05,372 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:05,374 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:18:07,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:07,655 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:18:07,657 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 97.114000082. 2021-12-16 14:18:07,657 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:18:07,660 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:18:07,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:18:08,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:18:08,173 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:18:08,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:18:08,198 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:18:08,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:18:08,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:18:08,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:18:08,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:18:08,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:18:08,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:08,482 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:18:08,483 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.825999975204. 2021-12-16 14:18:08,483 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:18:08,486 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:18:08,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:18:09,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:18:09,302 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:18:09,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:18:09,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:18:09,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:18:09,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:18:09,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:18:09,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:18:09,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:18:09,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:09,595 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:09,595 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:11,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:11,801 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:11,802 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:13,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:13,973 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:13,973 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:15,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:16,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:16,145 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:18,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:18,325 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:18,325 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:20,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:20,678 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:18:20,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:18:22,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:18:23,259 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:18:23,259 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 14.7760000229. ############################################################ # # RAM FootPrint: 3479.26171875 # ############################################################ 2021-12-16 14:18:23,289 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:18:23,289 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 75 seconds. Reason: Iteration:2 waiting to kill memc on all nodes 2021-12-16 14:19:38,293 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:19:38,295 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:19:43,296 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:19:43,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:44,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:19:44,424 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:44,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:19:44,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:19:44,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:19:44,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:19:44,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:19:44,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:19:44,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:19:44,719 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:19:44,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:19:44,812 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['570 memcached\n'], []) 2021-12-16 14:19:44,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:19:44,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:19:44,986 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['570 memcached\n'], []) 2021-12-16 14:19:44,986 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:19:44,990 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:19:45,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:45,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:19:45,523 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:45,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:19:45,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:19:45,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:19:45,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:19:45,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:19:45,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:19:45,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:19:45,648 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:19:45,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:19:45,720 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['199994 memcached\n'], []) 2021-12-16 14:19:45,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:19:45,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:19:45,842 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['199994 memcached\n'], []) 2021-12-16 14:19:45,842 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:19:45,845 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:19:46,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:46,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:19:46,417 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:46,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:19:46,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:19:46,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:19:46,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:19:46,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:19:46,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:19:46,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:19:46,569 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:19:46,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:19:46,638 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['39525 memcached\n'], []) 2021-12-16 14:19:46,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:19:46,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:19:46,803 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['39525 memcached\n'], []) 2021-12-16 14:19:46,803 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:19:46,806 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:19:46,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:47,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:19:47,216 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:47,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:19:47,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:19:47,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:19:47,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:19:47,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:19:47,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:19:47,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:19:47,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:19:47,351 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:19:47,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:19:47,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:47,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:47,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:47,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:47,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:47,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:47,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:47,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:47,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:47,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:47,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:47,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:47,755 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:47,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:47,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:47,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:47,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:47,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:47,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:48,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:48,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:48,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:48,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:48,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:48,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:48,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:48,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:48,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:48,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:48,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:48,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:48,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:48,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:49,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:49,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:49,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:49,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:49,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:49,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:49,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:49,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:49,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:49,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:49,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:49,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:50,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:50,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:19:50,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:19:50,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:19:50,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:19:50,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:19:50,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:19:50,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:19:50,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:19:50,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:19:50,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:19:50,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:50,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:19:50,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:19:50,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:19:50,846 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:19:50,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:19:50,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:50,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:19:50,888 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:19:50,888 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:19:50,891 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:19:51,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:52,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:19:52,184 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:52,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:19:52,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:19:52,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:19:52,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:19:52,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:19:52,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:19:52,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:19:52,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:19:52,328 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:19:52,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:19:52,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:52,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:52,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:19:52,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:19:52,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:52,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:52,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:52,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:52,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:52,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:52,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:52,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:52,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:52,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:52,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:52,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:52,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:52,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:53,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:53,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,104 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:53,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:53,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:53,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:53,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:53,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:53,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:53,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:53,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:53,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:53,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:53,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:53,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:53,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:53,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:53,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:54,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:54,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:54,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:54,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:54,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:54,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:54,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:19:54,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:19:54,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:19:54,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:19:54,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:19:54,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:19:54,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:19:54,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:19:54,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:54,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:19:54,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:19:54,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:19:54,880 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:19:54,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:19:54,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:54,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:19:54,917 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:19:54,918 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:19:54,921 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:19:55,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:55,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:19:55,454 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:55,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:19:55,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:19:55,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:19:55,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:19:55,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:19:55,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:19:55,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:19:55,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:19:55,619 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:19:55,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:19:55,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:55,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:55,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:19:55,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 14:19:55,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:55,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:55,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:19:55,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:19:55,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:55,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:19:56,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:19:56,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:19:56,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:19:56,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:19:56,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:19:56,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:19:56,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:19:56,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:56,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:19:56,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:56,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:19:56,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:56,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:19:56,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:56,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:56,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:19:56,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:56,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:57,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:19:57,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:57,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:19:57,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:57,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:19:57,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:57,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:19:57,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:57,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:19:57,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:57,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:19:57,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:57,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:19:57,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:57,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:57,908 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:19:57,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:57,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:58,022 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:58,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:19:58,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:58,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:19:58,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:58,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:19:58,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:19:58,224 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:19:58,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:19:58,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:19:58,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:19:58,302 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:19:58,302 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:19:58,302 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:19:58,305 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:19:58,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:19:58,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:19:58,836 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:19:58,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:19:58,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:19:58,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:19:58,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:19:58,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:19:58,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:19:58,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:19:58,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:19:59,157 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:19:59,161 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:19:59,161 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:01,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:01,332 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:01,336 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:01,336 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:03,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:03,517 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:03,523 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:03,523 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:05,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:05,707 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:05,711 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:05,711 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:07,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:07,881 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:07,887 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:07,887 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:09,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:10,055 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:10,061 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:10,062 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:12,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:12,244 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:12,247 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:12,249 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:14,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:14,427 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:14,430 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:14,431 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:16,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:16,594 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:16,599 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:16,601 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:18,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:18,772 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:20:18,776 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:18,776 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:20,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:21,007 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:21,007 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:23,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:23,160 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:23,161 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:25,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:25,315 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:25,315 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:27,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:27,473 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:27,473 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:29,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:29,618 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:29,619 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:31,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:31,782 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:31,783 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:33,783 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:33,930 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:33,930 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:35,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:36,071 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:36,072 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:38,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:38,226 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:38,226 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:40,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:40,387 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:40,388 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:42,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:42,536 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:42,536 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:44,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:44,704 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:44,704 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:46,704 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:46,865 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:46,865 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:48,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:49,023 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:49,023 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:51,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:51,183 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:51,184 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:53,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:53,348 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:53,348 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:55,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:55,509 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:55,509 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:57,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:57,657 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:57,657 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:20:59,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:20:59,819 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:20:59,819 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:01,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:01,979 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:01,980 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:03,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:04,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:04,144 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:06,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:06,313 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:06,315 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:08,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:08,470 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:08,470 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:10,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:10,634 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:10,634 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:12,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:12,792 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:12,792 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:14,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:14,956 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:14,956 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:16,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:17,121 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:17,121 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:19,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:19,963 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:19,963 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:21,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:22,180 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:22,180 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:24,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:24,358 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:24,358 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:26,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:26,536 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:26,536 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:28,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:28,720 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:28,721 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:30,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:30,904 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:21:30,904 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:21:32,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:33,851 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:21:33,852 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 95.5499999523. 2021-12-16 14:21:33,854 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:21:33,855 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:21:33,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:21:34,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:21:34,377 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:21:34,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:21:34,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:21:34,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:21:34,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:21:34,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:21:34,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:21:34,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:21:34,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:34,680 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:21:34,680 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.826999902725. 2021-12-16 14:21:34,680 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:21:34,684 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:21:34,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:21:35,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:21:35,154 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:21:35,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:21:35,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:21:35,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:21:35,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:21:35,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:21:35,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:21:35,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:21:35,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:21:35,457 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:21:35,457 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.776999950409. ############################################################ # # RAM FootPrint: 3438.91796875 # ############################################################ 2021-12-16 14:21:35,492 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:21:35,492 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:3 waiting to kill memc on all nodes 2021-12-16 14:23:27,496 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:23:27,496 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:23:32,497 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:23:32,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:32,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:23:33,026 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:33,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:23:33,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:23:33,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:23:33,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:23:33,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:23:33,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:23:33,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:23:33,174 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:23:33,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:23:33,243 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['2509 memcached\n'], []) 2021-12-16 14:23:33,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:23:33,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:23:33,335 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['2509 memcached\n'], []) 2021-12-16 14:23:33,335 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:23:33,338 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:23:33,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:33,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:23:33,855 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:33,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:23:33,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:23:33,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:23:33,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:23:33,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:23:33,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:23:33,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:23:33,976 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:23:33,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:23:34,059 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['200673 memcached\n'], []) 2021-12-16 14:23:34,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:23:34,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:23:34,190 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['200673 memcached\n'], []) 2021-12-16 14:23:34,190 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:23:34,193 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:23:34,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:34,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:23:34,740 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:34,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:23:34,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:23:34,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:23:34,783 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:23:34,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:23:34,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:23:34,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:23:34,852 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:23:34,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:23:34,905 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['41175 memcached\n'], []) 2021-12-16 14:23:34,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:23:34,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:23:34,990 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['41175 memcached\n'], []) 2021-12-16 14:23:34,990 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:23:34,993 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:23:35,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:35,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:23:35,354 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:35,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:23:35,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:23:35,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:23:35,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:23:35,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:23:35,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:23:35,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:23:35,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:23:35,516 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:23:35,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:23:35,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:35,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:35,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:35,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:35,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:35,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:35,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:35,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:35,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:35,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:35,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:35,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:36,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:36,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:36,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:36,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:36,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:36,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:36,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:36,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:36,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:36,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:36,645 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:36,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:36,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:36,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:36,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:36,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:36,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:36,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:36,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:37,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:37,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:37,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:37,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:37,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:37,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:37,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:37,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:23:37,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:23:37,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:23:37,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:23:37,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:23:37,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:23:37,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:23:37,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:23:37,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:37,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:23:37,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:23:37,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:23:37,792 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:23:37,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:23:37,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:37,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:23:37,835 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:23:37,836 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:23:37,838 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:23:37,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:38,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:23:38,334 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:38,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:23:38,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:23:38,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:23:38,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:23:38,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:23:38,433 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:23:38,448 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:23:38,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:23:38,482 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:23:38,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:23:38,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:38,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:38,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:23:38,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:23:38,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:38,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:38,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:38,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:38,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:38,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:38,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:38,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:38,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:39,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:39,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:39,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:39,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:39,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:39,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:39,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:39,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:39,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:39,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:39,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:39,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:39,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:39,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:39,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:39,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:39,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:39,979 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:40,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:40,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:40,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:40,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:40,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:40,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:40,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:40,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:40,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:40,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:23:40,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:23:40,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:23:40,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:23:40,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:23:40,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:23:40,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:40,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:23:40,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:23:40,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:23:40,790 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:23:40,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:23:40,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:40,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:23:40,832 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:23:40,832 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:23:40,835 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:23:40,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:41,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:23:41,348 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:41,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:23:41,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:23:41,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:23:41,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:23:41,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:23:41,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:23:41,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:23:41,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:23:41,467 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:23:41,502 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:23:41,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:23:41,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:23:41,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:23:41,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:23:41,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:23:41,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:23:41,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:23:41,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:23:41,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:23:41,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:23:41,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:41,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:41,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:42,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:23:42,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:42,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:23:42,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:42,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:23:42,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:42,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:23:42,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:42,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:23:42,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:42,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:23:42,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:42,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:23:42,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:42,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:23:42,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:42,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:23:42,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:42,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:23:42,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:42,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:42,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:42,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:23:43,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:43,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:43,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:43,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:23:43,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:43,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:43,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:43,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:23:43,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:43,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:43,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:43,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:23:43,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:43,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:23:43,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:43,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:23:43,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:23:43,480 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:23:43,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:23:43,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:23:43,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:23:43,522 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:23:43,522 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:23:43,522 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:23:43,525 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:23:43,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:23:43,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:23:44,040 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:23:44,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:23:44,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:23:44,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:23:44,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:23:44,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:23:44,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:23:44,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:23:44,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:44,365 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:44,372 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:44,372 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:46,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:46,516 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:46,519 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:46,519 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:48,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:48,687 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:48,693 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:48,693 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:50,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:50,857 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:50,861 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:50,861 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:52,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:53,039 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:53,043 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:53,043 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:55,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:55,211 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:55,216 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:55,216 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:57,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:57,401 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:57,405 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:57,405 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:23:59,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:23:59,582 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:23:59,586 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:23:59,586 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:01,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:01,770 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:24:01,775 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:01,775 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:03,775 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:03,956 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:24:03,960 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:03,960 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:05,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:06,141 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:24:06,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:06,144 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:08,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:08,290 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:08,292 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:10,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:10,443 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:10,443 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:12,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:12,592 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:12,594 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:14,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:14,746 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:14,746 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:16,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:16,904 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:16,904 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:18,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:19,055 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:19,055 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:21,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:21,201 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:21,201 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:23,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:23,346 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:23,346 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:25,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:25,517 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:25,519 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:27,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:27,683 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:27,684 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:29,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:29,845 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:29,845 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:31,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:32,009 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:32,010 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:34,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:34,171 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:34,173 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:36,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:36,334 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:36,334 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:38,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:38,492 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:38,493 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:40,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:40,651 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:40,651 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:42,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:42,809 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:42,809 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:44,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:44,967 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:44,967 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:46,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:47,127 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:47,128 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:49,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:49,282 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:49,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:51,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:51,443 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:51,444 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:53,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:53,612 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:53,614 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:55,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:55,770 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:55,772 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:57,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:24:57,930 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:24:57,930 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:24:59,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:00,088 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:00,088 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:02,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:02,244 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:02,244 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:04,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:04,407 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:04,407 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:06,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:06,686 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:06,686 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:08,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:08,878 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:08,880 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:10,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:11,052 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:11,052 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:13,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:13,224 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:13,224 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:15,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:15,401 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:15,401 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:17,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:17,569 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:25:17,569 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:25:19,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:20,003 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:25:20,003 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.4809999466. 2021-12-16 14:25:20,003 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:25:20,005 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:25:20,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:25:20,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:25:20,566 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:25:20,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:25:20,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:25:20,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:25:20,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:25:20,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:25:20,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:25:20,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:25:20,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:20,930 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:25:20,930 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.927000045776. 2021-12-16 14:25:20,930 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:25:20,934 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:25:21,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:25:21,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:25:21,509 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:25:21,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:25:21,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:25:21,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:25:21,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:25:21,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:25:21,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:25:21,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:25:21,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:25:21,882 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:25:21,884 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.953000068665. ############################################################ # # RAM FootPrint: 3386.52734375 # ############################################################ 2021-12-16 14:25:21,917 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:25:21,917 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 96 seconds. Reason: Iteration:4 waiting to kill memc on all nodes 2021-12-16 14:26:57,921 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:26:57,921 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:27:02,923 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:27:03,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:03,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:27:03,476 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:03,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:27:03,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:27:03,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:27:03,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:27:03,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:27:03,571 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:27:03,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:27:03,602 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:27:03,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:27:03,680 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['4414 memcached\n'], []) 2021-12-16 14:27:03,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:27:03,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:27:03,821 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['4414 memcached\n'], []) 2021-12-16 14:27:03,821 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:27:03,823 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:27:03,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:04,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:27:04,246 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:04,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:27:04,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:27:04,285 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:27:04,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:27:04,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:27:04,341 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:27:04,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:27:04,372 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:27:04,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:27:04,426 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['202695 memcached\n'], []) 2021-12-16 14:27:04,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:27:04,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:27:04,569 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['202695 memcached\n'], []) 2021-12-16 14:27:04,569 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:27:04,572 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:27:04,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:04,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:27:05,032 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:05,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:27:05,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:27:05,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:27:05,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:27:05,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:27:05,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:27:05,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:27:05,180 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:27:05,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:27:05,252 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['42766 memcached\n'], []) 2021-12-16 14:27:05,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:27:05,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:27:05,380 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['42766 memcached\n'], []) 2021-12-16 14:27:05,380 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:27:05,382 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:27:05,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:05,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:27:05,811 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:05,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:27:05,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:27:05,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:27:05,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:27:05,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:27:05,908 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:27:05,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:27:05,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:27:05,957 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:27:05,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:27:06,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:06,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:06,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:06,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:06,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:06,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:06,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:06,579 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:06,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:06,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:06,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:06,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:06,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:06,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:06,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:06,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:06,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:07,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:07,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:07,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:07,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:07,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:07,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:07,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:07,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:07,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:07,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:07,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:07,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:07,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:07,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:07,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:07,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:07,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:07,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:27:07,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:27:08,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:08,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:08,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:27:08,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:27:08,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:08,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:08,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:27:08,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:27:08,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:08,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:08,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:27:08,346 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:27:08,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:08,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:08,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:27:08,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:27:08,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:27:08,480 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:27:08,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:27:08,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:08,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:27:08,528 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:27:08,528 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:27:08,530 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:27:08,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:08,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:27:09,025 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:09,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:27:09,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:27:09,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:27:09,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:27:09,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:27:09,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:27:09,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:27:09,181 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:27:09,200 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:27:09,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:27:09,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:27:09,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:27:09,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:09,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:09,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:09,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:09,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:09,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:09,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:09,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:09,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:09,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:09,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:09,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:10,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:10,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:10,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:10,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:10,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:10,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:10,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:10,346 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:10,367 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:10,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:10,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:10,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:10,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,591 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:10,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:10,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:10,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:10,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:10,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:10,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:10,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:10,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:10,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:10,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:11,038 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:27:11,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:27:11,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:11,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:11,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:27:11,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:27:11,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:11,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:11,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:27:11,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:27:11,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:11,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:11,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:27:11,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:27:11,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:27:11,427 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:27:11,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:27:11,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:11,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:27:11,471 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:27:11,473 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:27:11,473 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:27:11,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:11,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:27:12,019 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:12,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:27:12,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:27:12,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:27:12,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:27:12,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:27:12,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:27:12,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:27:12,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:27:12,167 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:27:12,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:27:12,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:27:12,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:27:12,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:27:12,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:27:12,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:27:12,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:27:12,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:27:12,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:27:12,727 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:27:12,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:27:12,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:12,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:12,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:12,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:27:12,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:13,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:27:13,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:13,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:27:13,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:13,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:27:13,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:13,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:27:13,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:13,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:27:13,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,433 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:13,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:27:13,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:13,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:27:13,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:13,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:27:13,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:13,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:27:13,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:13,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:27:13,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:13,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:27:13,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:13,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:13,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:14,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:27:14,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:14,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:14,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:14,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:27:14,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:14,181 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:27:14,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:14,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:27:14,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:27:14,266 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:27:14,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:27:14,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:27:14,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:27:14,336 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:27:14,336 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:27:14,336 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:27:14,339 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:27:14,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:27:14,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:27:14,864 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:27:14,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:27:14,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:27:14,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:27:14,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:27:14,943 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:27:14,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:27:14,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:27:14,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:15,125 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:15,130 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:15,130 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:17,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:17,289 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:17,295 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:17,295 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:19,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:19,464 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:19,469 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:19,469 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:21,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:21,647 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:21,651 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:21,651 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:23,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:23,809 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:23,815 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:23,815 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:25,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:25,983 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:25,986 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:25,986 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:27,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:28,164 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:28,170 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:28,170 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:30,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:30,345 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:30,349 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:30,349 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:32,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:32,473 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:32,479 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:32,479 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:34,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:34,661 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:34,665 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:34,667 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:36,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:36,844 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:36,848 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:36,848 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:38,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:39,019 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:27:39,023 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:39,023 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:41,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:41,167 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:41,167 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:43,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:43,325 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:43,325 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:45,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:45,482 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:45,482 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:47,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:47,641 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:47,641 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:49,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:49,789 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:49,789 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:51,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:51,947 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:51,947 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:53,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:54,095 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:54,095 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:56,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:56,243 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:56,243 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:27:58,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:27:58,467 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:27:58,467 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:00,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:00,630 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:00,630 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:02,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:02,786 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:02,788 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:04,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:04,944 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:04,944 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:06,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:07,102 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:07,104 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:09,104 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:09,256 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:09,256 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:11,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:11,413 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:11,413 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:13,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:13,581 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:13,582 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:15,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:15,746 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:15,746 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:17,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:17,904 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:17,904 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:19,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:20,061 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:20,062 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:22,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:22,233 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:22,233 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:24,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:24,395 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:24,397 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:26,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:26,559 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:26,559 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:28,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:28,719 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:28,719 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:30,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:30,887 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:30,887 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:32,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:33,052 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:33,052 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:35,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:35,206 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:35,206 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:37,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:37,364 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:37,365 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:39,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:39,523 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:39,525 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:41,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:41,808 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:41,809 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:43,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:44,397 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:44,397 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:46,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:46,592 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:46,592 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:48,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:48,756 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:48,757 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:50,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:51,002 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:28:51,002 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:28:53,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:53,265 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:28:53,266 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.9290001392. 2021-12-16 14:28:53,266 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:28:53,269 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:28:53,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:28:53,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:28:53,783 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:28:53,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:28:53,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:28:53,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:28:53,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:28:53,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:28:53,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:28:53,908 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:28:53,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:54,121 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:28:54,121 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.855999946594. 2021-12-16 14:28:54,122 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:28:54,125 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:28:54,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:28:54,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:28:54,608 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:28:54,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:28:54,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:28:54,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:28:54,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:28:54,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:28:54,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:28:54,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:28:54,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:28:54,851 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:28:54,851 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.729000091553. ############################################################ # # RAM FootPrint: 3333.23046875 # ############################################################ 2021-12-16 14:28:54,882 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:28:54,882 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 107 seconds. Reason: Iteration:5 waiting to kill memc on all nodes 2021-12-16 14:30:41,888 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:30:41,888 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:30:46,891 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:30:47,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:47,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:30:47,401 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:47,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:30:47,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:30:47,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:30:47,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:30:47,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:30:47,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:30:47,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:30:47,506 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:30:47,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:30:47,565 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['6257 memcached\n'], []) 2021-12-16 14:30:47,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:30:47,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:30:47,654 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['6257 memcached\n'], []) 2021-12-16 14:30:47,654 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:30:47,657 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:30:47,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:48,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:30:48,086 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:48,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:30:48,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:30:48,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:30:48,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:30:48,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:30:48,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:30:48,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:30:48,198 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:30:48,198 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:30:48,249 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['203915 memcached\n'], []) 2021-12-16 14:30:48,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:30:48,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:30:48,381 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['203915 memcached\n'], []) 2021-12-16 14:30:48,381 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:30:48,385 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:30:48,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:48,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:30:48,726 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:48,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:30:48,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:30:48,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:30:48,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:30:48,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:30:48,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:30:48,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:30:48,865 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:30:48,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:30:48,917 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['44351 memcached\n'], []) 2021-12-16 14:30:48,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:30:48,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:30:49,065 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['44351 memcached\n'], []) 2021-12-16 14:30:49,066 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:30:49,069 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:30:49,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:49,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:30:49,520 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:49,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:30:49,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:30:49,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:30:49,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:30:49,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:30:49,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:30:49,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:30:49,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:30:49,694 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:30:49,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:30:49,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:49,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:49,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:49,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:49,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:49,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:49,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:50,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:50,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:50,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:50,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:50,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:50,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:50,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:50,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:50,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:50,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:50,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:50,752 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:50,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:50,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:50,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:50,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:50,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:50,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:51,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:51,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:51,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:51,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:51,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:51,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:51,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:51,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:51,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:51,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,683 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:51,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:51,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:30:51,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:30:51,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:51,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:51,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:30:51,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:30:52,022 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:52,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:52,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:30:52,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:30:52,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:52,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:52,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:30:52,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:30:52,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:52,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:52,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:30:52,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:30:52,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:30:52,384 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:30:52,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:30:52,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:52,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:30:52,431 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:30:52,431 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:30:52,434 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:30:52,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:52,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:30:52,931 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:52,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:30:52,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:30:52,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:30:52,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:30:53,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:30:53,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:30:53,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:30:53,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:30:53,065 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:30:53,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:30:53,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:30:53,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:30:53,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:53,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:53,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:53,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:53,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:53,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:53,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:53,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:53,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:53,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:53,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:53,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:53,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:53,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:53,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:53,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:53,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:54,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:54,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:54,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:54,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:54,252 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:54,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:54,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:54,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:54,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:54,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:54,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:54,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:54,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:54,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:54,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:54,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:30:54,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:30:54,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:30:54,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:30:54,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:54,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:54,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:30:54,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:30:55,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:55,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:55,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:30:55,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:30:55,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:30:55,096 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:30:55,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:30:55,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:55,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:30:55,145 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:30:55,145 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:30:55,148 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:30:55,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:55,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:30:55,687 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:55,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:30:55,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:30:55,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:30:55,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:30:55,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:30:55,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:30:55,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:30:55,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:30:55,834 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:30:55,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:30:55,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:55,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:55,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:30:56,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:30:56,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,066 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:30:56,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:30:56,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:30:56,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:30:56,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:30:56,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:30:56,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:30:56,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:30:56,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:56,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:30:56,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:56,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:30:56,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:56,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:30:56,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:56,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:30:56,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:56,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:56,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:57,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:30:57,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:57,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:30:57,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:57,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:30:57,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:57,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:30:57,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:57,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:30:57,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:57,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:30:57,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:57,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:30:57,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:57,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:30:57,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:57,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:30:57,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:57,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:30:57,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:30:57,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:57,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:30:57,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:30:57,931 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:30:57,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:30:57,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:30:57,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:30:57,984 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:30:57,984 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:30:57,984 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:30:57,989 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:30:58,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:30:58,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:30:58,512 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:30:58,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:30:58,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:30:58,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:30:58,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:30:58,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:30:58,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:30:58,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:30:58,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:30:58,821 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:30:58,826 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:30:58,826 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:00,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:00,996 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:01,002 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:01,002 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:03,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:03,184 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:03,188 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:03,188 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:05,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:05,367 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:05,371 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:05,371 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:07,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:07,556 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:07,561 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:07,561 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:09,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:09,763 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:09,766 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:09,766 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:11,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:11,878 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:11,882 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:11,882 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:13,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:14,069 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:14,072 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:14,073 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:16,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:16,246 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:16,250 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:16,252 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:18,252 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:18,430 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:18,434 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:18,436 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:20,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:20,617 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:20,621 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:20,621 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:22,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:22,798 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:22,802 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:22,802 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:24,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:24,973 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:24,976 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:24,976 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:26,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:27,144 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:27,148 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:27,148 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:29,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:29,332 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:31:29,335 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:29,335 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:31,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:31,486 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:31,486 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:33,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:33,645 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:33,645 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:35,645 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:35,831 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:35,831 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:37,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:38,003 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:38,003 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:40,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:40,180 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:40,181 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:42,181 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:42,342 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:42,342 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:44,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:44,529 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:44,529 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:46,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:46,676 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:46,676 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:48,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:48,851 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:48,851 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:50,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:51,010 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:51,010 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:53,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:53,165 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:53,165 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:55,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:55,328 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:55,328 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:57,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:57,493 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:57,493 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:31:59,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:31:59,651 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:31:59,651 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:01,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:01,808 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:01,808 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:03,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:03,970 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:03,970 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:05,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:06,128 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:06,128 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:08,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:08,289 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:08,289 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:10,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:10,450 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:10,450 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:12,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:12,615 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:12,615 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:14,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:14,772 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:14,772 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:16,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:16,940 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:16,940 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:18,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:19,104 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:19,105 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:21,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:21,263 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:21,263 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:23,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:23,438 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:23,438 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:25,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:25,676 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:25,676 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:27,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:28,079 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:28,079 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:30,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:30,299 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:30,299 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:32,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:32,467 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:32,467 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:34,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:34,640 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:34,641 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:36,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:36,805 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:36,805 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:38,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:38,976 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:38,976 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:40,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:41,141 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:41,141 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:43,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:43,301 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:32:43,302 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:32:45,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:45,897 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:32:45,897 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 107.912000179. 2021-12-16 14:32:45,897 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:32:45,900 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:32:46,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:32:46,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:32:46,410 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:32:46,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:32:46,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:32:46,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:32:46,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:32:46,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:32:46,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:32:46,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:32:46,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:46,688 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:32:46,688 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.791999816895. 2021-12-16 14:32:46,688 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:32:46,691 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:32:46,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:32:47,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:32:47,180 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:32:47,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:32:47,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:32:47,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:32:47,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:32:47,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:32:47,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:32:47,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:32:47,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:32:47,500 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:32:47,500 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.811000108719. ############################################################ # # RAM FootPrint: 3600.73828125 # ############################################################ 2021-12-16 14:32:47,535 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:32:47,535 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 84 seconds. Reason: Iteration:6 waiting to kill memc on all nodes 2021-12-16 14:34:11,540 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:34:11,542 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:34:16,543 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:34:16,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:17,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:34:17,163 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:17,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:34:17,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:34:17,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:34:17,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:34:17,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:34:17,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:34:17,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:34:17,332 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:34:17,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:34:17,387 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['8128 memcached\n'], []) 2021-12-16 14:34:17,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:34:17,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:34:17,517 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['8128 memcached\n'], []) 2021-12-16 14:34:17,517 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:34:17,522 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:34:17,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:17,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:34:18,025 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:18,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:34:18,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:34:18,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:34:18,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:34:18,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:34:18,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:34:18,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:34:18,141 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:34:18,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:34:18,216 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['205250 memcached\n'], []) 2021-12-16 14:34:18,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:34:18,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:34:18,361 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['205250 memcached\n'], []) 2021-12-16 14:34:18,362 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:34:18,365 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:34:18,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:18,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:34:18,841 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:18,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:34:18,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:34:18,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:34:18,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:34:18,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:34:18,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:34:18,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:34:19,023 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:34:19,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:34:19,114 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['45959 memcached\n'], []) 2021-12-16 14:34:19,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:34:19,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:34:19,260 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['45959 memcached\n'], []) 2021-12-16 14:34:19,262 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:34:19,263 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:34:19,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:19,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:34:19,671 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:19,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:34:19,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:34:19,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:34:19,737 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:34:19,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:34:19,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:34:19,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:34:19,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:34:19,825 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:34:19,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:34:19,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:19,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:19,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:20,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:20,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:20,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:20,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:20,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:20,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:20,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:20,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:20,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:20,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:20,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:20,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:20,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:20,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:20,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:20,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:20,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:20,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:20,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:20,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:21,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:21,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:21,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:21,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:21,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:21,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:21,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:21,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:21,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:21,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,571 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:21,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:21,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:34:21,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:34:21,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:34:21,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:34:21,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:21,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:21,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:34:22,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:34:22,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:22,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:22,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:34:22,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:34:22,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:22,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:22,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:34:22,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:34:22,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:34:22,332 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:34:22,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:34:22,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:22,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:34:22,387 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:34:22,387 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:34:22,391 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:34:22,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:22,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:34:22,901 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:22,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:34:22,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:34:22,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:34:22,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:34:22,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:34:22,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:34:23,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:34:23,020 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:34:23,036 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:34:23,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:34:23,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:23,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:23,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:23,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:23,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:23,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:23,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:23,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:23,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:23,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:23,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:23,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:23,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:23,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:23,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:23,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:23,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:24,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:24,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:24,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:24,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:24,331 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:24,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:24,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:24,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:24,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:24,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:24,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:24,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:24,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:24,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:24,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:24,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:24,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:34:24,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:34:24,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:24,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:25,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:34:25,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:34:25,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:25,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:25,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:34:25,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:34:25,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:25,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:25,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:34:25,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:34:25,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:25,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:25,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:34:25,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:34:25,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:34:25,423 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:34:25,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:34:25,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:25,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:34:25,470 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:34:25,470 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:34:25,471 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:34:25,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:25,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:34:25,976 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:25,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:34:25,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:34:26,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:34:26,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:34:26,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:34:26,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:34:26,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:34:26,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:34:26,128 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:34:26,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:34:26,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:34:26,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:34:26,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:34:26,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:34:26,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:34:26,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:34:26,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:34:26,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:34:26,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:34:26,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:34:26,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:26,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:34:26,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:26,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:26,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:26,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:34:26,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:27,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:34:27,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:27,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:34:27,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:27,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:34:27,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:27,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:34:27,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,489 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:27,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:34:27,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:27,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:34:27,681 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:27,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:34:27,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:27,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:27,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:27,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:34:27,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:28,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:28,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:34:28,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:28,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:28,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:34:28,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:28,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:28,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:34:28,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:28,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:28,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:34:28,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:34:28,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:28,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:34:28,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:34:28,551 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:34:28,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:34:28,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:34:28,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:34:28,596 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:34:28,596 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:34:28,598 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:34:28,601 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:34:28,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:34:29,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:34:29,131 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:34:29,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:34:29,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:34:29,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:34:29,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:34:29,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:34:29,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:34:29,252 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:34:29,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:29,428 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:29,434 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:29,434 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:31,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:31,586 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:31,594 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:31,594 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:33,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:33,766 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:33,770 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:33,770 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:35,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:35,940 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:35,944 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:35,944 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:37,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:38,127 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:38,131 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:38,131 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:40,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:40,316 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:40,321 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:40,321 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:42,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:42,483 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:42,489 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:42,489 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:44,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:44,674 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:44,680 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:44,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:46,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:46,871 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:46,875 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:46,875 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:48,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:49,049 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:49,055 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:49,055 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:51,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:51,224 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:34:51,229 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:51,230 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:53,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:53,375 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:53,377 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:55,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:55,536 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:55,536 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:57,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:57,683 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:57,683 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:34:59,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:34:59,829 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:34:59,829 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:01,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:01,980 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:01,980 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:03,982 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:04,137 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:04,137 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:06,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:06,282 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:06,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:08,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:08,423 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:08,423 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:10,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:10,588 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:10,588 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:12,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:12,743 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:12,743 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:14,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:14,911 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:14,911 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:16,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:17,072 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:17,073 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:19,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:19,236 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:19,236 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:21,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:21,394 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:21,394 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:23,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:23,555 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:23,555 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:25,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:25,721 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:25,723 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:27,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:27,871 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:27,871 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:29,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:30,032 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:30,032 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:32,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:32,188 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:32,190 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:34,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:34,351 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:34,351 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:36,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:36,513 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:36,513 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:38,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:38,674 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:38,674 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:40,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:40,839 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:40,839 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:42,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:42,994 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:42,994 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:44,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:45,158 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:45,158 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:47,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:47,319 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:47,319 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:49,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:49,476 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:49,476 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:51,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:51,634 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:51,634 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:53,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:53,799 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:53,799 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:55,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:56,117 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:56,117 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:35:58,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:35:58,299 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:35:58,299 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:36:00,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:36:00,842 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:36:00,842 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:36:02,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:36:03,046 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:36:03,046 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 94.4489998817. 2021-12-16 14:36:03,046 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:36:03,049 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:36:03,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:36:03,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:36:03,611 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:36:03,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:36:03,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:36:03,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:36:03,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:36:03,698 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:36:03,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:36:03,732 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:36:03,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:36:03,910 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:36:03,911 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.864000082016. 2021-12-16 14:36:03,911 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:36:03,914 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:36:04,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:36:04,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:36:04,259 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:36:04,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:36:04,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:36:04,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:36:04,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:36:04,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:36:04,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:36:04,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:36:04,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:36:04,500 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:36:04,500 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.588000059128. ############################################################ # # RAM FootPrint: 3569.21484375 # ############################################################ 2021-12-16 14:36:04,535 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:36:04,535 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 97 seconds. Reason: Iteration:7 waiting to kill memc on all nodes 2021-12-16 14:37:41,538 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:37:41,539 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:37:46,540 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:37:46,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:47,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:37:47,096 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:47,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:37:47,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:37:47,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:37:47,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:37:47,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:37:47,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:37:47,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:37:47,247 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:37:47,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:37:47,322 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['10053 memcached\n'], []) 2021-12-16 14:37:47,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:37:47,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:37:47,448 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['10053 memcached\n'], []) 2021-12-16 14:37:47,450 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:37:47,451 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:37:47,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:47,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:37:47,937 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:47,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:37:47,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:37:47,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:37:47,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:37:48,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:37:48,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:37:48,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:37:48,061 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:37:48,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:37:48,112 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['206465 memcached\n'], []) 2021-12-16 14:37:48,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:37:48,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:37:48,253 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['206465 memcached\n'], []) 2021-12-16 14:37:48,253 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:37:48,256 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:37:48,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:48,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:37:48,732 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:48,732 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:37:48,755 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:37:48,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:37:48,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:37:48,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:37:48,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:37:48,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:37:48,890 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:37:48,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:37:48,970 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['47535 memcached\n'], []) 2021-12-16 14:37:48,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:37:49,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:37:49,111 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['47535 memcached\n'], []) 2021-12-16 14:37:49,111 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:37:49,114 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:37:49,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:49,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:37:49,502 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:49,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:37:49,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:37:49,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:37:49,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:37:49,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:37:49,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:37:49,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:37:49,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:37:49,648 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:37:49,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:37:49,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:49,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:49,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:49,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:49,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:49,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:49,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:49,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:49,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:49,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:50,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:50,063 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:50,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:50,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:50,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:50,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:50,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:50,331 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:50,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:50,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:50,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:50,573 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:50,655 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:50,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:50,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:50,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:50,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:50,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:50,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:50,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:51,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:51,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:51,181 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:51,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:51,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:51,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:37:51,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:37:51,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:37:51,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:37:51,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:37:51,752 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:37:51,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:37:51,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:37:51,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:51,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:51,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:37:51,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:37:52,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:52,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:52,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:37:52,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:37:52,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:37:52,095 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:37:52,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:37:52,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:52,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:37:52,144 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:37:52,144 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:37:52,148 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:37:52,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:52,579 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:37:52,654 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:52,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:37:52,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:37:52,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:37:52,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:37:52,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:37:52,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:37:52,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:37:52,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:37:52,798 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:37:52,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:37:52,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:52,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:52,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:37:52,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:37:52,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:53,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:53,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:53,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:53,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:53,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:53,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:53,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:53,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:53,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:53,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:53,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:53,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:53,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:53,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:53,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:53,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:53,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:53,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,854 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:53,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:53,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:53,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:53,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:53,966 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:53,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:54,022 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:54,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:54,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:54,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:54,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:54,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:37:54,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:37:54,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:37:54,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:37:54,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:37:54,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:37:54,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:37:54,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:37:54,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:54,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:37:54,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:37:54,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:37:54,671 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:37:54,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:37:54,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:54,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:37:54,713 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:37:54,713 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:37:54,716 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:37:54,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:55,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:37:55,227 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:55,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:37:55,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:37:55,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:37:55,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:37:55,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:37:55,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:37:55,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:37:55,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:37:55,346 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:37:55,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:37:55,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:37:55,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 14:37:55,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:37:55,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:37:55,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:37:55,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:37:55,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:37:55,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:37:55,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:37:55,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:37:55,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:55,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:55,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:37:55,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:37:55,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:56,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:37:56,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:56,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:37:56,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:56,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:37:56,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:56,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:37:56,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:56,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:37:56,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,346 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,367 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:56,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:37:56,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:56,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:37:56,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:56,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:37:56,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:56,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:37:56,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,655 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:56,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:37:56,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,816 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:56,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:37:56,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:56,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:56,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:56,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:37:56,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:57,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:57,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:57,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:37:57,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:57,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:37:57,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:57,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:37:57,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:37:57,203 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:37:57,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:37:57,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:37:57,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:37:57,243 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:37:57,243 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:37:57,243 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:37:57,247 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:37:57,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:37:57,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:37:57,763 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:37:57,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:37:57,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:37:57,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:37:57,816 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:37:57,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:37:57,857 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:37:57,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:37:57,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:37:58,078 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:37:58,085 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:37:58,085 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:00,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:00,250 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:00,256 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:00,257 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:02,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:02,421 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:02,426 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:02,426 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:04,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:04,592 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:04,596 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:04,598 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:06,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:06,780 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:06,785 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:06,785 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:08,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:08,957 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:08,963 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:08,963 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:10,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:11,128 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:11,132 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:11,132 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:13,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:13,302 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:13,306 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:13,306 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:15,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:15,473 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:15,476 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:15,476 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:17,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:17,641 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:17,645 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:17,645 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:19,645 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:19,809 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:38:19,813 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:19,813 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:21,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:22,000 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:22,000 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:24,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:24,153 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:24,153 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:26,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:26,306 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:26,308 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:28,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:28,459 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:28,460 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:30,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:30,611 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:30,611 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:32,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:32,769 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:32,769 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:34,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:34,927 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:34,927 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:36,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:37,082 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:37,082 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:39,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:39,232 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:39,232 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:41,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:41,388 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:41,390 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:43,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:43,551 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:43,552 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:45,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:45,706 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:45,706 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:47,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:47,868 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:47,869 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:49,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:50,029 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:50,029 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:52,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:52,188 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:52,188 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:54,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:54,345 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:54,345 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:56,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:56,520 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:56,520 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:38:58,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:38:58,686 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:38:58,686 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:00,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:00,845 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:00,845 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:02,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:03,012 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:03,012 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:05,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:05,161 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:05,161 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:07,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:07,313 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:07,315 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:09,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:09,482 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:09,483 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:11,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:11,638 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:11,638 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:13,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:13,798 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:13,799 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:15,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:16,032 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:16,032 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:18,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:18,214 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:18,214 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:20,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:20,381 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:20,381 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:22,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:22,552 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:22,552 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:24,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:24,729 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:24,729 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:26,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:26,921 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:26,923 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:28,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:29,092 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:29,092 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:31,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:31,262 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:39:31,262 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:39:33,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:33,428 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:39:33,428 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.1849999428. 2021-12-16 14:39:33,428 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:39:33,431 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:39:33,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:39:33,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:39:33,937 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:39:33,938 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:39:33,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:39:33,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:39:34,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:39:34,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:39:34,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:39:34,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:39:34,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:34,288 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:39:34,288 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.859000205994. 2021-12-16 14:39:34,289 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:39:34,290 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:39:34,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:39:34,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:39:34,743 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:39:34,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:39:34,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:39:34,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:39:34,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:39:34,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:39:34,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:39:34,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:39:34,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:39:35,160 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:39:35,160 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.871000051498. ############################################################ # # RAM FootPrint: 3522.48828125 # ############################################################ 2021-12-16 14:39:35,190 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:39:35,190 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 68 seconds. Reason: Iteration:8 waiting to kill memc on all nodes 2021-12-16 14:40:43,194 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:40:43,194 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:40:48,196 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:40:48,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:49,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:40:49,713 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:49,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:40:49,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:40:49,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:40:49,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:40:49,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:40:49,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:40:49,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:40:49,947 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:40:49,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:40:50,022 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['11896 memcached\n'], []) 2021-12-16 14:40:50,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:40:50,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:40:50,236 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['11896 memcached\n'], []) 2021-12-16 14:40:50,236 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:40:50,240 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:40:50,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:50,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:40:50,770 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:50,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:40:50,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:40:50,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:40:50,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:40:50,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:40:50,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:40:50,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:40:50,900 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:40:50,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:40:50,951 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['208012 memcached\n'], []) 2021-12-16 14:40:50,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:40:51,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:40:51,094 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['208012 memcached\n'], []) 2021-12-16 14:40:51,094 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:40:51,096 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:40:51,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:51,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:40:51,571 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:51,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:40:51,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:40:51,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:40:51,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:40:51,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:40:51,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:40:51,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:40:51,726 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:40:51,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:40:51,788 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['49110 memcached\n'], []) 2021-12-16 14:40:51,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:40:51,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:40:51,944 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['49110 memcached\n'], []) 2021-12-16 14:40:51,944 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:40:51,947 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:40:52,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:52,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:40:52,421 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:52,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:40:52,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:40:52,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:40:52,477 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:40:52,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:40:52,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:40:52,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:40:52,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:40:52,571 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:40:52,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:40:52,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:52,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:52,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:52,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:52,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:52,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:52,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:52,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:52,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:52,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:52,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:52,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:52,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:52,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:53,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:53,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:53,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:53,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:53,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:53,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:53,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:53,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:53,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:53,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:53,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:53,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:53,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:53,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:53,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:53,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,854 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:53,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:53,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:53,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:53,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:54,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:54,038 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:54,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:54,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:54,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:54,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,316 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:40:54,341 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:40:54,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:40:54,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:40:54,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:40:54,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:40:54,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:40:54,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:40:54,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:54,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:40:54,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:40:54,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:40:54,766 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:40:54,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:40:54,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:54,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:40:54,819 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:40:54,819 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:40:54,822 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:40:54,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:55,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:40:55,267 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:55,267 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:40:55,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:40:55,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:40:55,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:40:55,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:40:55,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:40:55,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:40:55,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:40:55,420 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:40:55,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:40:55,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:55,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:55,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:40:55,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:40:55,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:55,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:55,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:55,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:55,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:55,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:55,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:55,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:55,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:55,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:55,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:55,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:56,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:56,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:56,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:56,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:56,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:56,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:56,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:56,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:56,316 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:56,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:56,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:56,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:56,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:56,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:56,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:56,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:56,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:56,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:56,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:56,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:56,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:56,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:56,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:56,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:56,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:57,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:57,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:57,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:57,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:57,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:40:57,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:40:57,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:57,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:57,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:40:57,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:40:57,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:57,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:57,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:40:57,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:40:57,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:57,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:57,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:40:57,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:40:57,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:40:57,326 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:40:57,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:40:57,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:57,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:40:57,365 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:40:57,365 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:40:57,368 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:40:57,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:40:57,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:40:57,894 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:40:57,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:40:57,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:40:57,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:40:57,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:40:57,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:40:57,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:40:58,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:40:58,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:40:58,039 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:40:58,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:40:58,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:40:58,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:40:58,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:40:58,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:40:58,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:40:58,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:40:58,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:40:58,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:40:58,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:40:58,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:40:58,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:58,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:40:58,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:58,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:40:58,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:58,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:40:58,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:58,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:40:58,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:58,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:58,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:58,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:40:58,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:59,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:40:59,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:59,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:40:59,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:59,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:40:59,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:59,267 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:40:59,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:59,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:40:59,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:59,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:40:59,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:59,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:40:59,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:59,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:40:59,627 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:59,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:40:59,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:40:59,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:59,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:40:59,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:40:59,747 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:40:59,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:40:59,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:40:59,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:40:59,779 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:40:59,779 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:40:59,779 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:40:59,782 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:40:59,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:41:00,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:41:00,282 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:41:00,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:41:00,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:41:00,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:41:00,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:41:00,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:41:00,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:41:00,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:41:00,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:00,661 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-12-16 14:41:00,667 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:00,667 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:02,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:02,826 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:02,832 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:02,834 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:04,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:04,999 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:05,005 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:05,005 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:07,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:07,184 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:07,190 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:07,190 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:09,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:09,358 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:09,362 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:09,362 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:11,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:11,535 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:11,539 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:11,539 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:13,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:13,706 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:13,710 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:13,710 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:15,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:15,880 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:15,884 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:15,884 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:17,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:18,028 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:18,035 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:18,035 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:20,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:20,197 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:20,200 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:20,200 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:22,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:22,390 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:22,395 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:22,395 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:24,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:24,562 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:41:24,565 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:24,565 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:26,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:26,713 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:26,713 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:28,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:28,865 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:28,865 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:30,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:31,013 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:31,013 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:33,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:33,167 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:33,167 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:35,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:35,323 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:35,323 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:37,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:37,464 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:37,464 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:39,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:39,608 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:39,608 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:41,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:41,759 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:41,759 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:43,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:43,918 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:43,920 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:45,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:46,085 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:46,085 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:48,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:48,246 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:48,246 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:50,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:50,404 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:50,404 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:52,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:52,562 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:52,562 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:54,563 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:54,723 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:54,723 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:56,724 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:56,887 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:56,887 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:41:58,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:41:59,040 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:41:59,042 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:01,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:01,201 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:01,201 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:03,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:03,359 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:03,359 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:05,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:05,522 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:05,522 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:07,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:07,667 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:07,667 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:09,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:09,826 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:09,828 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:11,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:11,983 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:11,983 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:13,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:14,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:14,144 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:16,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:16,299 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:16,299 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:18,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:18,454 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:18,454 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:20,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:20,609 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:20,609 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:22,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:22,767 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:22,769 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:24,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:24,934 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:24,936 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:26,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:27,095 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:27,095 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:29,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:30,148 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:30,148 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:32,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:32,464 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:32,464 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:34,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:35,276 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:42:35,276 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:42:37,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:37,446 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:42:37,446 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 97.6660001278. 2021-12-16 14:42:37,446 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:42:37,448 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:42:37,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:42:37,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:42:37,987 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:42:37,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:42:38,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:42:38,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:42:38,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:42:38,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:42:38,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:42:38,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:42:38,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:38,282 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:42:38,283 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.837999820709. 2021-12-16 14:42:38,285 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:42:38,286 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:42:38,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:42:38,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:42:38,900 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:42:38,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:42:38,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:42:38,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:42:38,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:42:38,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:42:39,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:42:39,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:42:39,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:42:39,250 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:42:39,250 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.964999914169. ############################################################ # # RAM FootPrint: 3487.66015625 # ############################################################ 2021-12-16 14:42:39,286 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:42:39,286 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 112 seconds. Reason: Iteration:9 waiting to kill memc on all nodes 2021-12-16 14:44:31,289 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:44:31,289 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:44:36,292 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:44:36,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:36,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:44:36,780 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:36,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:44:36,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:44:36,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:44:36,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:44:36,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:44:36,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:44:36,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:44:36,914 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:44:36,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:44:36,971 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['13784 memcached\n'], []) 2021-12-16 14:44:36,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:44:37,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:44:37,096 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['13784 memcached\n'], []) 2021-12-16 14:44:37,096 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:44:37,099 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:44:37,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:37,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:44:37,561 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:37,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:44:37,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:44:37,591 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:44:37,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:44:37,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:44:37,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:44:37,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:44:37,664 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:44:37,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:44:37,723 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['209295 memcached\n'], []) 2021-12-16 14:44:37,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:44:37,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:44:37,865 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['209295 memcached\n'], []) 2021-12-16 14:44:37,865 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:44:37,868 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:44:37,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:38,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:44:38,345 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:38,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:44:38,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:44:38,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:44:38,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:44:38,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:44:38,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:44:38,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:44:38,519 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:44:38,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:44:38,605 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['50656 memcached\n'], []) 2021-12-16 14:44:38,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:44:38,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:44:38,747 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['50656 memcached\n'], []) 2021-12-16 14:44:38,747 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:44:38,750 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:44:38,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:39,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:44:39,206 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:39,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:44:39,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:44:39,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:44:39,270 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:44:39,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:44:39,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:44:39,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:44:39,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:44:39,364 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:44:39,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:44:39,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:39,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:39,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:39,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:39,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:39,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:39,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:39,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:39,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:39,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:39,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:39,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:39,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:39,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:39,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:39,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:39,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:39,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:39,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:39,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:39,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,038 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:40,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:40,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:40,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:40,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:40,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:40,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,367 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:40,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:40,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:40,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:40,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:40,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:40,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:40,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:40,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:40,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:40,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:40,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:40,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:40,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:40,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:40,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:41,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:41,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:44:41,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:44:41,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:41,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:44:41,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:44:41,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:41,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:44:41,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:44:41,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:41,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:44:41,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:44:41,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:41,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:44:41,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:44:41,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:44:41,361 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:44:41,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:44:41,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:41,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:44:41,400 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:44:41,401 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:44:41,403 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:44:41,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:41,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:44:41,911 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:41,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:44:41,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:44:41,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:44:41,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:44:41,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:44:42,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:44:42,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:44:42,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:44:42,051 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:44:42,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:44:42,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:44:42,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:44:42,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:42,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:42,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:42,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:42,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:42,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:42,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:42,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:42,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:42,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:42,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:42,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:42,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:43,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:43,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:43,198 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:43,224 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:43,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:43,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:43,477 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:43,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:43,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:43,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:43,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:43,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:43,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:43,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:43,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:43,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:43,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:44,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:44,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:44,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:44,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:44,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:44:44,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:44:44,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:44:44,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:44:44,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:44:44,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:44:44,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:44,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:44:44,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:44:44,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:44:44,750 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:44:44,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:44:44,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:44,775 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:44:44,785 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:44:44,785 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:44:44,789 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:44:44,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:45,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:44:45,869 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:45,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:44:45,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:44:45,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:44:45,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:44:45,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:44:45,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:44:46,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:44:46,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:44:46,051 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:44:46,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:44:46,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:44:46,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:44:46,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:44:46,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:44:46,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:44:46,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:44:46,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:44:46,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:44:46,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:44:46,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:44:46,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:46,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:44:46,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:46,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:44:46,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:46,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:44:46,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:46,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:44:46,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:46,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:44:46,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:46,946 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:46,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:46,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:44:47,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:47,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:44:47,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:47,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:44:47,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:47,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:44:47,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:47,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:44:47,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:47,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:44:47,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:47,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:44:47,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:47,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:44:47,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:47,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:44:47,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:44:47,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:47,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:44:47,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:44:47,862 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:44:47,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:44:47,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:44:47,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:44:47,915 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:44:47,915 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:44:47,915 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:44:47,917 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:44:48,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:44:48,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:44:48,385 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:44:48,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:44:48,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:44:48,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:44:48,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:44:48,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:44:48,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:44:48,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:44:48,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:48,746 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:48,753 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:48,753 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:44:50,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:50,910 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:50,914 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:50,914 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:44:52,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:53,092 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:53,098 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:53,098 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:44:55,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:55,250 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:55,255 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:55,255 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:44:57,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:57,420 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:57,426 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:57,426 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:44:59,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:44:59,598 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:44:59,601 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:44:59,601 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:01,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:01,756 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:01,760 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:01,760 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:03,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:03,938 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:03,943 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:03,943 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:05,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:06,115 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:06,118 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:06,118 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:08,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:08,283 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:08,289 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:08,290 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:10,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:10,394 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:10,398 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:10,398 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:12,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:12,558 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:12,562 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:12,562 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:14,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:14,733 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:14,736 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:14,736 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:16,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:16,862 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:16,867 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:16,867 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:18,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:19,016 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:45:19,019 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:19,020 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:21,020 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:21,230 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:21,230 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:23,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:23,375 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:23,375 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:25,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:25,528 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:25,528 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:27,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:27,683 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:27,683 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:29,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:29,844 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:29,844 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:31,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:32,009 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:32,010 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:34,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:34,170 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:34,170 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:36,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:36,319 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:36,319 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:38,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:38,473 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:38,473 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:40,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:40,664 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:40,664 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:42,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:42,822 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:42,822 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:44,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:44,980 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:44,980 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:46,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:47,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:47,154 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:49,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:49,315 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:49,315 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:51,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:51,470 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:51,470 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:53,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:53,634 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:53,634 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:55,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:55,786 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:55,786 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:57,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:45:57,941 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:45:57,941 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:45:59,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:00,104 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:00,105 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:02,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:02,259 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:02,260 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:04,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:04,420 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:04,420 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:06,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:06,585 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:06,585 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:08,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:08,739 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:08,739 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:10,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:11,390 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:11,391 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:13,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:13,578 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:13,578 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:15,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:15,763 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:15,763 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:17,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:17,950 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:17,950 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:19,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:20,121 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:20,122 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:22,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:22,295 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:22,295 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:24,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:24,467 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:24,469 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:26,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:26,661 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:26,661 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:28,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:28,829 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:28,829 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:30,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:31,019 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:31,019 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:33,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:33,194 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:46:33,194 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:46:35,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:35,381 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:46:35,381 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 107.46600008. 2021-12-16 14:46:35,381 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:46:35,384 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:46:35,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:46:35,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:46:35,901 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:46:35,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:46:35,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:46:35,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:46:35,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:46:35,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:46:36,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:46:36,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:46:36,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:36,209 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:46:36,210 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.828000068665. 2021-12-16 14:46:36,210 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:46:36,213 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:46:36,331 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:46:36,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:46:36,696 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:46:36,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:46:36,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:46:36,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:46:36,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:46:36,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:46:36,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:46:36,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:46:36,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:46:37,042 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:46:37,042 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.832000017166. ############################################################ # # RAM FootPrint: 3419.05859375 # ############################################################ 2021-12-16 14:46:37,072 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:46:37,072 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 118 seconds. Reason: Iteration:10 waiting to kill memc on all nodes 2021-12-16 14:48:35,078 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:48:35,078 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:48:40,081 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:48:40,204 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:40,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:48:40,575 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:40,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:48:40,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:48:40,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:48:40,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:48:40,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:48:40,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:48:40,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:48:40,667 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:48:40,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:48:40,723 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['15580 memcached\n'], []) 2021-12-16 14:48:40,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:48:40,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:48:40,809 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['15580 memcached\n'], []) 2021-12-16 14:48:40,809 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:48:40,812 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:48:40,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:41,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:48:41,269 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:41,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:48:41,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:48:41,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:48:41,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:48:41,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:48:41,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:48:41,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:48:41,503 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:48:41,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:48:41,559 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['210532 memcached\n'], []) 2021-12-16 14:48:41,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:48:41,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:48:41,667 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['210532 memcached\n'], []) 2021-12-16 14:48:41,667 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:48:41,671 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:48:41,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:42,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:48:42,196 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:42,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:48:42,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:48:42,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:48:42,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:48:42,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:48:42,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:48:42,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:48:42,355 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:48:42,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:48:42,423 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['52278 memcached\n'], []) 2021-12-16 14:48:42,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:48:42,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:48:42,578 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['52278 memcached\n'], []) 2021-12-16 14:48:42,578 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:48:42,579 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:48:42,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:42,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:48:43,029 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:43,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:48:43,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:48:43,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:48:43,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:48:43,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:48:43,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:48:43,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:48:43,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:48:43,184 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:48:43,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:48:43,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:43,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:43,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:43,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:43,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:43,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:43,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:43,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:43,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:43,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:43,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:43,966 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:43,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:44,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:44,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:44,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:44,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:44,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:44,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:44,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:44,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:44,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:44,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:44,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:44,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:44,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:44,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:44,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:44,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:44,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:44,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:44,946 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:44,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:45,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:45,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:48:45,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:48:45,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:48:45,341 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:48:45,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:48:45,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:48:45,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:48:45,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:48:45,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:45,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:48:45,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:48:45,704 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:48:45,720 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:48:45,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:48:45,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:45,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:48:45,765 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:48:45,766 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:48:45,767 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:48:45,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:46,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:48:46,286 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:46,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:48:46,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:48:46,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:48:46,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:48:46,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:48:46,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:48:46,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:48:46,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:48:46,426 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:48:46,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:48:46,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:46,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:46,563 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:48:46,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:48:46,617 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:46,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:46,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:46,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:46,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:46,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:46,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:46,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:46,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:46,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:46,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:46,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:47,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:47,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:47,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:47,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:47,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:47,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:47,285 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:47,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:47,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:47,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:47,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:47,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:47,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:47,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:47,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:47,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:47,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:47,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:47,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:47,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:47,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:47,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:47,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:47,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:47,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:48,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:48,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:48,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:48,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:48:48,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:48:48,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:48,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:48:48,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:48:48,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:48,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:48:48,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:48:48,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:48,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:48:48,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:48:48,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:48:48,490 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:48:48,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:48:48,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:48,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:48:48,539 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:48:48,539 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:48:48,542 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:48:48,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:48,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:48:49,076 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:49,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:48:49,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:48:49,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:48:49,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:48:49,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:48:49,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:48:49,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:48:49,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:48:49,203 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:48:49,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:48:49,252 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:48:49,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:48:49,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:48:49,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:48:49,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:48:49,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:48:49,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:48:49,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:48:49,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:48:49,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:48:49,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:49,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:48:49,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:49,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:48:49,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:49,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:49,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:49,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:48:49,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:50,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:48:50,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:50,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:48:50,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:50,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:48:50,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:50,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:48:50,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:50,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:48:50,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:50,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:48:50,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:50,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:48:50,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,589 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:50,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:48:50,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:50,681 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:48:50,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:50,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:48:50,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:50,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:48:50,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:48:50,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:50,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:48:50,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:48:50,892 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:48:50,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:48:50,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:48:50,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:48:50,926 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:48:50,926 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:48:50,926 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:48:50,928 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:48:51,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:48:51,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:48:51,424 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:48:51,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:48:51,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:48:51,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:48:51,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:48:51,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:48:51,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:48:51,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:48:51,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:48:51,655 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: 'ep_warmup_thread' 2021-12-16 14:48:51,660 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:48:51,660 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:48:53,660 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:48:53,836 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:48:53,842 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:48:53,842 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:48:55,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:48:55,983 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:48:55,989 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:48:55,989 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:48:57,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:48:58,151 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:48:58,155 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:48:58,155 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:00,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:00,332 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:00,336 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:00,336 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:02,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:02,506 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:02,510 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:02,512 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:04,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:04,677 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:04,680 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:04,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:06,681 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:06,849 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:06,854 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:06,854 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:08,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:09,010 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:09,015 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:09,016 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:11,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:11,176 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:11,180 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:11,180 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:13,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:13,349 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:13,352 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:13,354 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:15,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:15,520 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:15,525 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:15,525 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:17,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:17,694 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:17,698 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:17,700 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:19,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:19,859 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:19,865 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:19,865 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:21,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:21,980 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:21,986 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:21,986 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:23,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:24,161 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:24,164 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:24,165 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:26,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:26,292 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:26,296 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:26,298 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:28,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:28,464 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:28,469 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:28,470 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:30,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:30,644 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:30,648 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:30,648 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:32,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:32,832 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:32,836 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:32,836 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:34,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:34,973 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:34,979 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:34,979 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:36,979 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:37,128 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:37,135 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:37,135 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:39,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:39,312 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:39,316 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:39,316 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:41,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:41,460 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:49:41,466 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:41,466 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:43,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:43,618 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:43,618 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:45,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:45,773 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:45,773 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:47,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:47,930 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:47,930 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:49,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:50,081 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:50,081 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:52,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:52,236 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:52,236 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:54,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:54,391 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:54,391 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:56,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:56,549 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:56,549 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:49:58,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:49:58,700 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:49:58,700 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:00,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:00,917 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:00,918 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:02,918 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:03,063 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:03,063 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:05,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:05,220 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:05,220 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:07,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:07,382 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:07,384 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:09,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:09,558 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:09,558 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:11,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:11,724 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:11,726 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:13,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:13,881 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:13,881 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:15,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:16,051 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:16,052 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:18,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:18,211 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:18,213 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:20,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:20,374 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:20,374 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:22,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:22,532 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:22,532 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:24,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:24,677 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:24,677 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:26,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:26,842 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:26,842 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:28,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:29,005 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:29,006 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:31,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:31,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:31,155 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:33,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:33,315 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:33,315 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:35,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:35,469 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:35,470 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:37,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:37,632 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:37,632 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:39,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:39,796 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:39,796 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:41,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:41,959 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:41,960 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:43,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:44,398 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:44,398 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:46,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:46,812 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:46,812 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:48,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:48,999 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:49,000 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:51,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:51,164 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:51,165 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:53,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:53,351 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:53,352 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:55,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:55,532 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:50:55,532 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:50:57,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:57,802 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:50:57,803 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 126.877999783. 2021-12-16 14:50:57,803 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:50:57,805 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:50:57,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:50:58,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:50:58,331 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:50:58,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:50:58,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:50:58,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:50:58,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:50:58,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:50:58,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:50:58,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:50:58,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:58,609 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:50:58,609 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.805999994278. 2021-12-16 14:50:58,611 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:50:58,612 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:50:59,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:50:59,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:50:59,407 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:50:59,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:50:59,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:50:59,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:50:59,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:50:59,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:50:59,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:50:59,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:50:59,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:50:59,811 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:50:59,811 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 1.20000004768. ############################################################ # # RAM FootPrint: 3765.484375 # ############################################################ 2021-12-16 14:50:59,842 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:50:59,842 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 66 seconds. Reason: Iteration:11 waiting to kill memc on all nodes 2021-12-16 14:52:05,846 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:52:05,848 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:52:10,848 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:52:11,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:12,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:52:12,545 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:12,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:52:12,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:52:12,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:52:12,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:52:12,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:52:12,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:52:12,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:52:12,667 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:52:12,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:52:12,736 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['17608 memcached\n'], []) 2021-12-16 14:52:12,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:52:13,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:52:13,596 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['17608 memcached\n'], []) 2021-12-16 14:52:13,596 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:52:13,601 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:52:13,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:14,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:52:14,114 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:14,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:52:14,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:52:14,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:52:14,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:52:14,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:52:14,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:52:14,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:52:14,260 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:52:14,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:52:14,342 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['212040 memcached\n'], []) 2021-12-16 14:52:14,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:52:14,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:52:14,489 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['212040 memcached\n'], []) 2021-12-16 14:52:14,490 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:52:14,493 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:52:14,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:14,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:52:14,956 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:14,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:52:14,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:52:14,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:52:15,022 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:52:15,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:52:15,066 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:52:15,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:52:15,108 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:52:15,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:52:15,194 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['53893 memcached\n'], []) 2021-12-16 14:52:15,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:52:15,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:52:15,342 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['53893 memcached\n'], []) 2021-12-16 14:52:15,342 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:52:15,345 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:52:15,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:15,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:52:15,799 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:15,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:52:15,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:52:15,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:52:15,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:52:15,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:52:15,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:52:15,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:52:15,938 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:52:15,957 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:52:15,982 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:52:16,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:16,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:16,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:16,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:16,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:16,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:16,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:16,448 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:16,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:16,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:16,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:16,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:16,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:16,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:16,816 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:16,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:16,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:16,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:16,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:16,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:16,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:17,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:17,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:17,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:17,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:17,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:17,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:17,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:17,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:17,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:17,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:17,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:17,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:52:17,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:52:17,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:52:17,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:52:17,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:52:17,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:52:17,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:52:17,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:52:17,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:17,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:52:17,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:52:17,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:52:17,831 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:52:17,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:52:17,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:17,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:52:17,884 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:52:17,884 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:52:17,888 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:52:18,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:18,316 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:52:18,387 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:18,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:52:18,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:52:18,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:52:18,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:52:18,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:52:18,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:52:18,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:52:18,499 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:52:18,513 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:52:18,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:52:18,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:18,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:18,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:18,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:18,683 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:18,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:18,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:18,755 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:18,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:18,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:18,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:18,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:18,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:18,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:18,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:18,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:18,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:18,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:18,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:18,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:18,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:19,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:19,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:19,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:19,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:19,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:19,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:19,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:19,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:19,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:19,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:19,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:19,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:19,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:19,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:19,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:19,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:19,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:19,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:19,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:19,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:19,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:19,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:52:20,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:52:20,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:20,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:20,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:52:20,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:52:20,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:20,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:20,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:52:20,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:52:20,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:20,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:20,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:52:20,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:52:20,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:20,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:20,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:52:20,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:52:20,502 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:52:20,519 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:52:20,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:52:20,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:20,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:52:20,565 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:52:20,565 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:52:20,568 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:52:20,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:20,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:52:21,079 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:21,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:52:21,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:52:21,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:52:21,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:52:21,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:52:21,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:52:21,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:52:21,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:52:21,209 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:52:21,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:52:21,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:52:21,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:52:21,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:52:21,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:52:21,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:52:21,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:52:21,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:52:21,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:52:21,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:52:21,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:52:21,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:21,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:52:21,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:21,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:21,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:22,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:52:22,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:22,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:52:22,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:22,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:52:22,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:22,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:52:22,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:22,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:52:22,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:22,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:52:22,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:22,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:52:22,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:22,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:52:22,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:22,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:22,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:22,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:52:22,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:23,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:23,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:52:23,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:23,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:23,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:52:23,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:23,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:23,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:52:23,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:23,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:23,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:52:23,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:52:23,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:23,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:52:23,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:52:23,497 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:52:23,499 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:52:23,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:52:23,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:52:23,555 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:52:23,555 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:52:23,555 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:52:23,559 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:52:23,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:52:23,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:52:24,032 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:52:24,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:52:24,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:52:24,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:52:24,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:52:24,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:52:24,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:52:24,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:52:24,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:24,253 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:24,257 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:24,259 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:26,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:26,381 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:26,385 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:26,385 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:28,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:28,543 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:28,548 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:28,549 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:30,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:30,717 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:30,720 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:30,720 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:32,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:32,904 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:32,908 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:32,908 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:34,908 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:35,075 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:35,078 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:35,078 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:37,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:37,250 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:37,255 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:37,255 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:39,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:39,443 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:39,447 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:39,447 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:41,448 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:41,615 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:41,619 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:41,619 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:43,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:43,779 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:43,783 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:43,783 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:45,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:45,943 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:45,950 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:45,950 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:47,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:48,135 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:52:48,141 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:48,141 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:50,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:50,282 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:50,282 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:52,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:52,437 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:52,437 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:54,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:54,595 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:54,595 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:56,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:56,746 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:56,746 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:52:58,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:52:58,910 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:52:58,911 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:00,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:01,066 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:01,066 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:03,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:03,223 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:03,223 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:05,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:05,369 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:05,369 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:07,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:07,556 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:07,556 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:09,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:09,711 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:09,713 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:11,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:11,880 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:11,880 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:13,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:14,048 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:14,048 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:16,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:16,213 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:16,214 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:18,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:18,374 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:18,375 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:20,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:20,535 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:20,535 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:22,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:22,700 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:22,700 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:24,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:24,859 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:24,861 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:26,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:27,013 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:27,013 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:29,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:29,173 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:29,173 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:31,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:31,335 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:31,335 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:33,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:33,499 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:33,499 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:35,499 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:35,657 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:35,658 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:37,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:37,819 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:37,819 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:39,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:39,987 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:39,987 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:41,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:42,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:42,145 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:44,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:44,306 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:44,306 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:46,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:47,138 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:47,138 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:49,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:49,392 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:49,394 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:51,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:51,556 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:51,558 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:53,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:53,779 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:53,779 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:55,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:55,959 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:55,959 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:53:57,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:53:58,130 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:53:58,130 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:54:00,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:54:00,311 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:54:00,312 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 96.756000042. 2021-12-16 14:54:00,312 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:54:00,313 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:54:00,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:54:00,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:54:00,864 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:54:00,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:54:00,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:54:00,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:54:00,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:54:00,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:54:00,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:54:00,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:54:00,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:54:01,158 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:54:01,160 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.848000049591. 2021-12-16 14:54:01,161 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:54:01,163 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:54:01,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:54:01,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:54:01,496 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:54:01,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:54:01,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:54:01,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:54:01,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:54:01,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:54:01,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:54:01,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:54:01,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:54:01,767 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:54:01,767 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.606999874115. ############################################################ # # RAM FootPrint: 3727.13671875 # ############################################################ 2021-12-16 14:54:01,799 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:54:01,799 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 62 seconds. Reason: Iteration:12 waiting to kill memc on all nodes 2021-12-16 14:55:03,806 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:55:03,806 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:55:08,809 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:55:08,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:12,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:55:13,124 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:13,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:55:13,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:55:13,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:55:13,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:55:13,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:55:13,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:55:13,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:55:13,321 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:55:13,321 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:55:13,400 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['19550 memcached\n'], []) 2021-12-16 14:55:13,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:55:13,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:55:13,632 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['19550 memcached\n'], []) 2021-12-16 14:55:13,632 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:55:13,637 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:55:13,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:14,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:55:14,184 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:14,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:55:14,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:55:14,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:55:14,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:55:14,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:55:14,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:55:14,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:55:14,328 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:55:14,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:55:14,392 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['213326 memcached\n'], []) 2021-12-16 14:55:14,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:55:14,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:55:14,536 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['213326 memcached\n'], []) 2021-12-16 14:55:14,536 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:55:14,539 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:55:14,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:14,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:55:14,963 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:14,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:55:14,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:55:15,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:55:15,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:55:15,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:55:15,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:55:15,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:55:15,132 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:55:15,132 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:55:15,213 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['55481 memcached\n'], []) 2021-12-16 14:55:15,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:55:15,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:55:15,332 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['55481 memcached\n'], []) 2021-12-16 14:55:15,334 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:55:15,335 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:55:15,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:15,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:55:15,793 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:15,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:55:15,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:55:15,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:55:15,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:55:15,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:55:15,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:55:15,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:55:15,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:55:15,954 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:55:15,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:55:16,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:16,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:16,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:16,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:16,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:16,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:16,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,477 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:16,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:16,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:16,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:16,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:16,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:16,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:16,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:16,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:16,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:16,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:16,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:16,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:17,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:17,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:17,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:17,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:17,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:17,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:17,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:17,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,591 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:17,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:17,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:17,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:17,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:55:17,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:55:17,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:17,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:17,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:55:17,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:55:18,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:18,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:18,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:55:18,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:55:18,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:18,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:18,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:55:18,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:55:18,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:18,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:18,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:55:18,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:55:18,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:18,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:18,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 14:55:18,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 14:55:18,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:55:18,401 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:55:18,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:55:18,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:18,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:55:18,446 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:55:18,447 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:55:18,450 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:55:18,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:18,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:55:18,953 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:18,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:55:18,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:55:18,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:55:19,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:55:19,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:55:19,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:55:19,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:55:19,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:55:19,086 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:55:19,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:55:19,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:55:19,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:55:19,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:19,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:19,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:19,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:19,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:19,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:19,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,732 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:19,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:19,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:19,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:19,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:19,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:19,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:20,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:20,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:20,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:20,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:20,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:20,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:20,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:20,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:20,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:20,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,683 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:20,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:20,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:20,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:20,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:20,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:20,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:20,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:20,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:21,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:21,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:55:21,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:55:21,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:55:21,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:55:21,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:55:21,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:55:21,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,573 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:55:21,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:55:21,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:21,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:55:21,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:55:21,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:55:21,701 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:55:21,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:55:21,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:21,727 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:55:21,740 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:55:21,740 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:55:21,743 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:55:21,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:22,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:55:22,273 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:22,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:55:22,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:55:22,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:55:22,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:55:22,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:55:22,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:55:22,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:55:22,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:55:22,397 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:55:22,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:55:22,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:55:22,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 14:55:22,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,551 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,573 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:55:22,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:55:22,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:55:22,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:55:22,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:55:22,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:55:22,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:55:22,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:55:22,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:55:22,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:55:22,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:22,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:55:22,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:22,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:22,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:22,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:55:22,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:23,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:55:23,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:23,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:55:23,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:23,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:55:23,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:23,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:55:23,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:23,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:55:23,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:23,489 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:55:23,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,591 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:23,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:55:23,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,681 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:23,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:55:23,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:23,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:55:23,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:23,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:23,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:24,020 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:55:24,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:24,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:24,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:24,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:55:24,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:24,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:55:24,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:24,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:55:24,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:55:24,318 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:55:24,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:55:24,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:55:24,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:55:24,381 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:55:24,381 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:55:24,381 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:55:24,384 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:55:24,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:55:24,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:55:24,897 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:55:24,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:55:24,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:55:24,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:55:24,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:55:24,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:55:24,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:55:25,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:55:25,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:25,150 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:25,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:25,154 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:27,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:27,332 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:27,335 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:27,335 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:29,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:29,515 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:29,519 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:29,519 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:31,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:31,680 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:31,684 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:31,684 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:33,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:33,854 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:33,861 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:33,861 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:35,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:36,045 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:36,049 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:36,049 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:38,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:38,229 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:38,233 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:38,233 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:40,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:40,395 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:55:40,400 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:40,400 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:42,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:42,555 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:42,555 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:44,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:44,709 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:44,710 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:46,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:46,874 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:46,874 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:48,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:49,045 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:49,045 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:51,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:51,198 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:51,198 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:53,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:53,346 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:53,348 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:55,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:55,500 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:55,500 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:57,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:57,653 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:57,653 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:55:59,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:55:59,818 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:55:59,818 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:01,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:01,984 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:01,986 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:03,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:04,148 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:04,150 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:06,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:06,309 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:06,309 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:08,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:08,469 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:08,469 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:10,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:10,632 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:10,632 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:12,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:12,788 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:12,789 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:14,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:14,947 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:14,947 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:16,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:17,107 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:17,108 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:19,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:19,270 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:19,270 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:21,272 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:21,434 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:21,434 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:23,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:23,595 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:23,595 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:25,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:25,753 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:25,753 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:27,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:27,908 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:27,908 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:29,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:30,086 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:30,086 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:32,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:32,239 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:32,239 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:34,239 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:34,400 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:34,400 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:36,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:36,552 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:36,553 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:38,553 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:38,714 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:38,716 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:40,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:40,878 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:40,878 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:42,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:43,040 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:43,042 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:45,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:45,207 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:45,207 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:47,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:47,857 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:47,858 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:49,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:50,214 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:50,214 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:56:52,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:52,770 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:56:52,770 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 88.388999939. 2021-12-16 14:56:52,772 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:56:52,775 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:56:52,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:56:53,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:56:53,302 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:56:53,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:56:53,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:56:53,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:56:53,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:56:53,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:56:53,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:56:53,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:56:53,448 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:53,634 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:56:53,635 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.864000082016. 2021-12-16 14:56:53,637 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:56:53,638 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:56:53,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:56:53,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:56:53,969 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:56:53,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:56:53,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:56:53,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:56:54,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:56:54,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:56:54,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:56:54,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:56:54,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:54,246 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:56:54,246 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.64 2021-12-16 14:56:56,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:56:56,670 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:56:56,671 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 3.03399991989. ############################################################ # # RAM FootPrint: 3680.54296875 # ############################################################ 2021-12-16 14:56:56,703 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 14:56:56,703 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:13 waiting to kill memc on all nodes 2021-12-16 14:58:30,707 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 14:58:30,707 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 14:58:35,710 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:58:36,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:36,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:58:36,920 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:36,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:58:36,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:58:36,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:58:36,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:58:37,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:58:37,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:58:37,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:58:37,615 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:58:37,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:58:37,713 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['21374 memcached\n'], []) 2021-12-16 14:58:37,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 14:58:37,783 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 14:58:37,898 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['21374 memcached\n'], []) 2021-12-16 14:58:37,898 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:58:37,901 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:58:38,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:38,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:58:38,424 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:38,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:58:38,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:58:38,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:58:38,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:58:38,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:58:38,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:58:38,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:58:38,543 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:58:38,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:58:38,596 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['214395 memcached\n'], []) 2021-12-16 14:58:38,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 14:58:38,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 14:58:38,729 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['214395 memcached\n'], []) 2021-12-16 14:58:38,730 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:58:38,733 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:58:38,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:39,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:58:39,184 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:39,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:58:39,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:58:39,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:58:39,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:58:39,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:58:39,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:58:39,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:58:39,359 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 14:58:39,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:58:39,446 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['57020 memcached\n'], []) 2021-12-16 14:58:39,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 14:58:39,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 14:58:39,584 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['57020 memcached\n'], []) 2021-12-16 14:58:39,584 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:58:39,588 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 14:58:39,704 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:39,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 14:58:40,052 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:40,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 14:58:40,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 14:58:40,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 14:58:40,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 14:58:40,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 14:58:40,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 14:58:40,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 14:58:40,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:58:40,204 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 14:58:40,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:58:40,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:40,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:40,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:40,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:40,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:40,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:40,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:40,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:40,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:40,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:40,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:40,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:40,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:40,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:40,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:41,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:41,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:41,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:41,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:41,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:41,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,283 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:41,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:41,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:41,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:41,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:41,573 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:41,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:41,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:41,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:41,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:41,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:41,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:41,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:41,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:41,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:42,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:58:42,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:58:42,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:42,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:42,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:58:42,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:58:42,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:42,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:42,198 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:58:42,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:58:42,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:42,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:42,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:58:42,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:58:42,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:42,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:42,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 14:58:42,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 14:58:42,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:58:42,454 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 14:58:42,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:58:42,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:42,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:58:42,505 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 14:58:42,506 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 14:58:42,507 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:58:42,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:42,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:58:43,006 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:43,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:58:43,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:58:43,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:58:43,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:58:43,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:58:43,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:58:43,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:58:43,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:58:43,157 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 14:58:43,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:58:43,204 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:58:43,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:58:43,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:43,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:43,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:43,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:43,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:43,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:43,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:43,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:43,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:43,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:43,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:44,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:44,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,084 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:44,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:44,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:44,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:44,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:44,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:44,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:44,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:44,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:44,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:44,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:44,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:44,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:44,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:44,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:44,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:44,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:45,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:45,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:45,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:45,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:45,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:45,404 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:58:45,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 14:58:45,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:58:45,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 14:58:45,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:58:45,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 14:58:45,816 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:45,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:58:45,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 14:58:45,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:58:45,938 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 14:58:45,938 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:58:45,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:45,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:58:45,986 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 14:58:45,986 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 14:58:45,989 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 14:58:46,107 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:46,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 14:58:46,492 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:46,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 14:58:46,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 14:58:46,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 14:58:46,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 14:58:46,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 14:58:46,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 14:58:46,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 14:58:46,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 14:58:46,607 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 14:58:46,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 14:58:46,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:46,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:46,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:58:46,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 14:58:46,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:46,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:46,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:58:46,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 14:58:46,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:46,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:46,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:58:46,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 14:58:46,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:46,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:46,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:58:46,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 14:58:46,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:46,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:46,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:58:47,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 14:58:47,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,038 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:47,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 14:58:47,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:47,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 14:58:47,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:47,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 14:58:47,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,293 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:47,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 14:58:47,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:47,385 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 14:58:47,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:47,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 14:58:47,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:47,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 14:58:47,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,627 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:47,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 14:58:47,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:47,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 14:58:47,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:47,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 14:58:47,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:47,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 14:58:47,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:47,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:47,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:47,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 14:58:48,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:48,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:48,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:48,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 14:58:48,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:48,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:48,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:48,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 14:58:48,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:48,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 14:58:48,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:48,181 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 14:58:48,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 14:58:48,206 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 14:58:48,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 14:58:48,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 14:58:48,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 14:58:48,250 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 14:58:48,250 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 14:58:48,250 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 14:58:48,255 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 14:58:48,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 14:58:48,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 14:58:48,750 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 14:58:48,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 14:58:48,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 14:58:48,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 14:58:48,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 14:58:48,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 14:58:48,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 14:58:48,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 14:58:48,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:49,030 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:49,035 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:49,035 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:58:51,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:51,197 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:51,201 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:51,201 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:58:53,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:53,358 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:53,362 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:53,362 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:58:55,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:55,532 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:55,539 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:55,539 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:58:57,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:57,726 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:57,730 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:57,730 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:58:59,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:58:59,913 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:58:59,917 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:58:59,917 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:01,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:02,086 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:02,094 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:02,094 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:04,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:04,269 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:04,273 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:04,273 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:06,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:06,448 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:06,453 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:06,453 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:08,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:08,634 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:08,638 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:08,638 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:10,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:10,819 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:10,825 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:10,825 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:12,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:12,999 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:13,003 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:13,003 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:15,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:15,174 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:15,177 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:15,177 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:17,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:17,348 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:17,352 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:17,352 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:19,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:19,519 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 14:59:19,523 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:19,525 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:21,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:21,671 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:21,671 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:23,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:23,895 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:23,895 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:25,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:26,056 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:26,056 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:28,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:28,214 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:28,214 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:30,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:30,364 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:30,364 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:32,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:32,506 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:32,507 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:34,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:34,651 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:34,651 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:36,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:36,851 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:36,851 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:38,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:39,019 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:39,019 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:41,020 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:41,177 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:41,177 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:43,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:43,345 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:43,345 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:45,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:45,510 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:45,510 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:47,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:47,671 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:47,673 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:49,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:49,829 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:49,829 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:51,829 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:51,992 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:51,993 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:53,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:54,160 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:54,161 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:56,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:56,328 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:56,328 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 14:59:58,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 14:59:58,490 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 14:59:58,490 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:00,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:00,655 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:00,657 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:02,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:02,819 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:02,819 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:04,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:04,986 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:04,986 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:06,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:07,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:07,154 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:09,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:09,980 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:09,980 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:11,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:12,244 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:12,244 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:14,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:14,421 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:14,421 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:16,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:16,595 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:16,596 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:18,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:19,371 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:19,371 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:21,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:21,542 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:21,543 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:23,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:23,733 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:23,734 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:25,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:25,967 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:25,967 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:27,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:28,164 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:28,164 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:30,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:30,348 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:00:30,348 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:00:32,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:37,424 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:00:37,424 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.173000097. 2021-12-16 15:00:37,424 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:00:37,427 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:00:37,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:00:37,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:00:37,953 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:00:37,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:00:37,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:00:37,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:00:38,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:00:38,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:00:38,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:00:38,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:00:38,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:38,305 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:00:38,306 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.882999897003. 2021-12-16 15:00:38,306 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:00:38,309 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:00:38,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:00:38,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:00:38,795 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:00:38,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:00:38,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:00:38,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:00:38,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:00:38,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:00:38,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:00:38,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:00:38,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:00:39,132 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:00:39,132 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.825999975204. ############################################################ # # RAM FootPrint: 3606.63671875 # ############################################################ 2021-12-16 15:00:39,173 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:00:39,174 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 103 seconds. Reason: Iteration:14 waiting to kill memc on all nodes 2021-12-16 15:02:22,178 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:02:22,178 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:02:27,181 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:02:27,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:27,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:02:27,733 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:27,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:02:27,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:02:27,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:02:27,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:02:27,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:02:27,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:02:27,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:02:27,865 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:02:27,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:02:27,924 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['23173 memcached\n'], []) 2021-12-16 15:02:27,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:02:27,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:02:28,016 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['23173 memcached\n'], []) 2021-12-16 15:02:28,016 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:02:28,020 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:02:28,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:28,341 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:02:28,421 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:28,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:02:28,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:02:28,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:02:28,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:02:28,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:02:28,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:02:28,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:02:28,546 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:02:28,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:02:28,621 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['215980 memcached\n'], []) 2021-12-16 15:02:28,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:02:28,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:02:28,729 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['215980 memcached\n'], []) 2021-12-16 15:02:28,729 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:02:28,732 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:02:28,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:29,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:02:29,190 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:29,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:02:29,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:02:29,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:02:29,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:02:29,285 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:02:29,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:02:29,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:02:29,344 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:02:29,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:02:29,414 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['58610 memcached\n'], []) 2021-12-16 15:02:29,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:02:29,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:02:29,552 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['58610 memcached\n'], []) 2021-12-16 15:02:29,552 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:02:29,556 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:02:29,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:29,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:02:30,009 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:30,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:02:30,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:02:30,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:02:30,063 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:02:30,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:02:30,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:02:30,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:02:30,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:02:30,145 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:02:30,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:02:30,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:30,283 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:30,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:30,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:30,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:30,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:30,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:30,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:30,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,579 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:30,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:30,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:30,700 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:30,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:30,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:30,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:30,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:30,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:30,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:30,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:30,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:30,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,017 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:31,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:31,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:31,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:31,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:31,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:31,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:31,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:31,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:31,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:31,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:31,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:31,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:02:31,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:02:31,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:02:31,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:02:31,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:31,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:31,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:02:32,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:02:32,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:32,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:32,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:02:32,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:02:32,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:32,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:32,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:02:32,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:02:32,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:02:32,188 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:02:32,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:02:32,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:32,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:02:32,233 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:02:32,233 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:02:32,236 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:02:32,359 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:32,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:02:32,746 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:32,747 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:02:32,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:02:32,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:02:32,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:02:32,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:02:32,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:02:32,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:02:32,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:02:32,892 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:02:32,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:02:32,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:32,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:02:33,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:02:33,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:33,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:33,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:33,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:33,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:33,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:33,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:33,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:33,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:33,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:33,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:33,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:33,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:33,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:34,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:34,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:34,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:34,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:34,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,410 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:34,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:34,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:34,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:34,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:34,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:34,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:34,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:34,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:34,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:34,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:34,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:35,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:35,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:35,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,214 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:35,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:35,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:02:35,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:02:35,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:02:35,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:02:35,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:02:35,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:02:35,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:35,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:02:35,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:02:35,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:02:35,743 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:02:35,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:02:35,755 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:35,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:02:35,782 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:02:35,782 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:02:35,786 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:02:35,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:36,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:02:36,289 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:36,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:02:36,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:02:36,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:02:36,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:02:36,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:02:36,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:02:36,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:02:36,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:02:36,420 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:02:36,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:02:36,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:36,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:36,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:02:36,563 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:02:36,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:36,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:36,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:02:36,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:02:36,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:36,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:36,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:02:36,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:02:36,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:36,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:36,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:02:36,979 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:02:37,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:02:37,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:02:37,107 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:37,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:02:37,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:37,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:02:37,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,321 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:37,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:02:37,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:37,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:02:37,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:37,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:02:37,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:37,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:02:37,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:37,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:02:37,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:37,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:37,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:38,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:02:38,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:38,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:02:38,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:38,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:02:38,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,331 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:38,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:02:38,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:38,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:02:38,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:38,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:02:38,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:38,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:02:38,821 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,848 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:02:38,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:38,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:02:38,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:02:38,936 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:02:38,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:02:38,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:02:38,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:02:38,994 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:02:38,994 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:02:38,994 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:02:38,999 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:02:39,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:02:39,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:02:39,523 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:02:39,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:02:39,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:02:39,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:02:39,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:02:39,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:02:39,627 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:02:39,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:02:39,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:39,884 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:39,891 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:39,891 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:41,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:42,068 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:42,072 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:42,072 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:44,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:44,252 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:44,256 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:44,536 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:46,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:46,717 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:46,720 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:46,720 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:48,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:48,881 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:48,887 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:48,887 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:50,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:51,056 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:51,061 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:51,062 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:53,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:53,240 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:53,246 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:53,246 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:55,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:55,414 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:55,421 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:55,421 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:57,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:57,605 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:57,611 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:57,611 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:02:59,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:02:59,792 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:02:59,796 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:02:59,796 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:01,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:01,967 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:01,970 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:01,970 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:03,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:04,140 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:04,144 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:04,144 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:06,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:06,249 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:06,253 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:06,253 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:08,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:08,415 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:08,418 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:08,420 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:10,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:10,592 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:10,596 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:10,596 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:12,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:12,769 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:12,775 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:12,775 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:14,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:14,951 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:14,956 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:14,956 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:16,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:17,078 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:17,082 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:17,082 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:19,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:19,260 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:19,265 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:19,265 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:21,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:21,415 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:21,421 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:21,421 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:23,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:23,601 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:23,605 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:23,605 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:25,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:25,776 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:03:25,780 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:25,782 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:27,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:27,930 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:27,930 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:29,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:30,085 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:30,085 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:32,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:32,253 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:32,253 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:34,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:34,417 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:34,417 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:36,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:36,575 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:36,575 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:38,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:38,723 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:38,723 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:40,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:40,871 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:40,871 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:42,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:43,025 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:43,025 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:45,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:45,173 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:45,173 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:47,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:47,351 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:47,351 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:49,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:49,545 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:49,545 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:51,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:51,749 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:51,749 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:53,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:53,923 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:53,923 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:55,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:56,075 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:56,076 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:03:58,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:03:58,243 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:03:58,243 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:00,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:00,407 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:00,407 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:02,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:02,572 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:02,572 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:04,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:04,733 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:04,733 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:06,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:06,897 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:06,897 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:08,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:09,053 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:09,053 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:11,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:11,207 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:11,207 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:13,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:13,371 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:13,371 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:15,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:15,532 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:15,532 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:17,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:17,687 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:17,687 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:19,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:19,848 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:19,848 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:21,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:22,022 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:22,023 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:24,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:24,187 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:24,187 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:26,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:26,342 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:26,342 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:28,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:28,512 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:28,513 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:30,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:30,693 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:30,693 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:32,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:32,868 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:32,868 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:34,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:35,032 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:35,032 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:37,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:37,217 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:37,217 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:39,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:39,405 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:39,405 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:41,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:41,598 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:04:41,599 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:04:43,599 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:43,780 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:04:43,780 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 124.786000013. 2021-12-16 15:04:43,782 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:04:43,785 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:04:43,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:04:44,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:04:44,313 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:04:44,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:04:44,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:04:44,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:04:44,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:04:44,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:04:44,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:04:44,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:04:44,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:44,651 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:04:44,651 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.868999958038. 2021-12-16 15:04:44,651 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:04:44,654 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:04:44,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:04:45,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:04:45,108 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:04:45,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:04:45,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:04:45,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:04:45,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:04:45,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:04:45,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:04:45,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:04:45,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:04:45,447 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:04:45,447 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.796000003815. ############################################################ # # RAM FootPrint: 4034.31640625 # ############################################################ 2021-12-16 15:04:45,483 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:04:45,483 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 93 seconds. Reason: Iteration:15 waiting to kill memc on all nodes 2021-12-16 15:06:18,487 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:06:18,489 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:06:23,490 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:06:23,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:23,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:06:23,887 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:23,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:06:23,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:06:23,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:06:24,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:06:24,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:06:24,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:06:24,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:06:24,082 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:06:24,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:06:24,144 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['25108 memcached\n'], []) 2021-12-16 15:06:24,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:06:24,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:06:24,303 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['25108 memcached\n'], []) 2021-12-16 15:06:24,305 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:06:24,308 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:06:24,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:24,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:06:24,786 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:24,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:06:24,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:06:24,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:06:24,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:06:24,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:06:24,862 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:06:24,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:06:24,891 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:06:24,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:06:24,944 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['217123 memcached\n'], []) 2021-12-16 15:06:24,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:06:24,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:06:25,049 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['217123 memcached\n'], []) 2021-12-16 15:06:25,049 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:06:25,052 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:06:25,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:25,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:06:25,515 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:25,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:06:25,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:06:25,558 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:06:25,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:06:25,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:06:25,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:06:25,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:06:25,663 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:06:25,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:06:25,717 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['60258 memcached\n'], []) 2021-12-16 15:06:25,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:06:25,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:06:25,839 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['60258 memcached\n'], []) 2021-12-16 15:06:25,841 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:06:25,844 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:06:25,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:26,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:06:26,253 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:26,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:06:26,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:06:26,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:06:26,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:06:26,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:06:26,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:06:26,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:06:26,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:06:26,411 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:06:26,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:06:26,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:26,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:26,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:26,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:26,619 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:26,660 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:26,698 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:26,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:26,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:26,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:26,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:26,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:26,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:26,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:26,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:26,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:26,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:27,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:27,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:27,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:27,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:27,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:27,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:27,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:27,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:27,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:27,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,553 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:27,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:27,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:27,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:27,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:27,806 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:27,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:27,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:27,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:27,918 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:27,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:27,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:27,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,078 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:28,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:28,145 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:06:28,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:06:28,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:06:28,410 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:06:28,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:06:28,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:06:28,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:06:28,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:06:28,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:28,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:06:28,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:06:28,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:06:28,773 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:06:28,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:06:28,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:28,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:06:28,819 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:06:28,819 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:06:28,822 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:06:28,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:29,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:06:29,332 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:29,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:06:29,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:06:29,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:06:29,390 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:06:29,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:06:29,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:06:29,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:06:29,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:06:29,471 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:06:29,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:06:29,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:29,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:29,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:06:29,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:06:29,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:29,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:29,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:29,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:29,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:29,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:29,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:29,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:29,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:29,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:30,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:30,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:30,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:30,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,229 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:30,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:30,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:30,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:30,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:30,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:30,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:30,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:30,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:30,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:30,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:30,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:30,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:30,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:30,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:31,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:31,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:31,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:31,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,173 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:31,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:31,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,242 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:31,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:31,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:31,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:31,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:06:31,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:06:31,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:06:31,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:06:31,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:06:31,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:06:31,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:06:31,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:06:31,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,727 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:31,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:06:31,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:06:31,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:06:31,782 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:06:31,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:06:31,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:31,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:06:31,822 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:06:31,822 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:06:31,825 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:06:31,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:32,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:06:32,323 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:32,323 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:06:32,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:06:32,352 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:06:32,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:06:32,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:06:32,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:06:32,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:06:32,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:06:32,441 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:06:32,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:06:32,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:32,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:32,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:06:32,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:06:32,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:32,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:32,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:06:32,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:06:32,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:32,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:32,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:06:32,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:06:32,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:32,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:32,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:06:32,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:06:33,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,049 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:06:33,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:06:33,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:33,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:06:33,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:33,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:06:33,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,437 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:33,499 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:06:33,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:33,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:06:33,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:33,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:06:33,732 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:33,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:06:33,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:33,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:33,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:33,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:06:33,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:34,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:06:34,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:34,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:06:34,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:34,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:06:34,219 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,265 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:34,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:06:34,298 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,316 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:34,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:06:34,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:34,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:06:34,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:34,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:06:34,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:06:34,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:34,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:06:34,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:06:34,614 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:06:34,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:06:34,627 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:06:34,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:06:34,667 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:06:34,667 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:06:34,667 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:06:34,671 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:06:34,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:06:35,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:06:35,125 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:06:35,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:06:35,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:06:35,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:06:35,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:06:35,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:06:35,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:06:35,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:06:35,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:35,444 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:35,448 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:35,448 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:37,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:37,618 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:37,621 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:37,621 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:39,622 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:39,799 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:39,802 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:39,802 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:41,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:41,960 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:41,963 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:41,964 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:43,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:44,137 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:44,141 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:44,141 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:46,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:46,313 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:46,318 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:46,318 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:48,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:48,483 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:48,487 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:48,487 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:50,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:50,668 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:50,673 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:50,673 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:52,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:52,835 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:52,841 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:52,841 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:54,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:54,989 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:54,993 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:54,993 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:56,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:57,164 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:57,168 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:57,168 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:06:59,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:06:59,352 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:06:59,358 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:06:59,358 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:01,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:01,519 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:07:01,523 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:01,523 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:03,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:03,707 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:07:03,710 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:03,711 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:05,711 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:05,885 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:07:05,890 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:05,891 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:07,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:08,052 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:08,053 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:10,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:10,207 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:10,207 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:12,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:12,361 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:12,361 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:14,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:14,516 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:14,516 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:16,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:16,674 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:16,674 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:18,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:18,823 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:18,823 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:20,823 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:20,976 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:20,976 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:22,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:23,121 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:23,121 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:25,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:25,302 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:25,302 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:27,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:27,470 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:27,470 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:29,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:29,628 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:29,628 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:31,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:31,786 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:31,786 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:33,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:33,940 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:33,941 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:35,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:36,092 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:36,092 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:38,094 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:38,256 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:38,256 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:40,256 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:40,421 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:40,421 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:42,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:42,571 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:42,572 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:44,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:44,726 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:44,726 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:46,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:46,877 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:46,878 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:48,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:49,026 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:49,026 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:51,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:51,193 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:51,193 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:53,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:53,357 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:53,357 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:55,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:55,519 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:55,519 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:57,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:57,680 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:57,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:07:59,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:07:59,851 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:07:59,851 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:01,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:02,013 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:02,015 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:04,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:04,177 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:04,177 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:06,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:06,631 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:06,632 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:08,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:08,793 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:08,795 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:10,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:10,990 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:10,990 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:12,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:13,153 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:13,153 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:15,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:15,487 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:15,487 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:17,489 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:17,822 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:08:17,822 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:08:19,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:20,153 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:08:20,154 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 105.486000061. 2021-12-16 15:08:20,154 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:08:20,157 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:08:20,270 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:08:20,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:08:20,687 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:08:20,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:08:20,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:08:20,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:08:20,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:08:20,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:08:20,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:08:20,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:08:20,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:21,016 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:08:21,016 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.862999916077. 2021-12-16 15:08:21,017 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:08:21,019 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:08:21,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:08:21,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:08:21,392 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:08:21,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:08:21,410 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:08:21,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:08:21,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:08:21,461 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:08:21,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:08:21,489 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:08:21,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:08:21,677 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:08:21,677 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.65900015831. ############################################################ # # RAM FootPrint: 3933.1875 # ############################################################ 2021-12-16 15:08:21,709 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:08:21,709 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:16 waiting to kill memc on all nodes 2021-12-16 15:09:55,713 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:09:55,713 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:10:00,716 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:10:00,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:01,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:10:01,351 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:01,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:10:01,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:10:01,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:10:01,477 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:10:01,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:10:01,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:10:01,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:10:01,562 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:10:01,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:10:01,644 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['27155 memcached\n'], []) 2021-12-16 15:10:01,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:10:01,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:10:01,855 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['27155 memcached\n'], []) 2021-12-16 15:10:01,855 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:10:01,858 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:10:01,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:02,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:10:02,368 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:02,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:10:02,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:10:02,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:10:02,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:10:02,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:10:02,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:10:02,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:10:02,503 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:10:02,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:10:02,576 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['218693 memcached\n'], []) 2021-12-16 15:10:02,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:10:02,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:10:02,707 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['218693 memcached\n'], []) 2021-12-16 15:10:02,707 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:10:02,710 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:10:02,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:03,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:10:03,194 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:03,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:10:03,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:10:03,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:10:03,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:10:03,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:10:03,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:10:03,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:10:03,377 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:10:03,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:10:03,463 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['61847 memcached\n'], []) 2021-12-16 15:10:03,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:10:03,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:10:03,601 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['61847 memcached\n'], []) 2021-12-16 15:10:03,601 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:10:03,604 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:10:03,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:03,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:10:04,055 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:04,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:10:04,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:10:04,104 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:10:04,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:10:04,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:10:04,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:10:04,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:10:04,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:10:04,224 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:10:04,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:10:04,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:04,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:04,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:04,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:04,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:04,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:04,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:04,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:04,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:04,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:04,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:04,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:04,983 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:05,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:05,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:05,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:05,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:05,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:05,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:05,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:05,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:05,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:05,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,591 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:05,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:05,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:05,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:05,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:05,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:05,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:05,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:05,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:05,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:06,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:06,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:06,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:10:06,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:10:06,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:10:06,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:10:06,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:10:06,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:10:06,367 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:10:06,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:10:06,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,454 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:06,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:10:06,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:10:06,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:10:06,519 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:10:06,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:10:06,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:06,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:10:06,558 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:10:06,558 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:10:06,561 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:10:06,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:06,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:10:07,035 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:07,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:10:07,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:10:07,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:10:07,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:10:07,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:10:07,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:10:07,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:10:07,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:10:07,174 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:10:07,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:10:07,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:07,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:07,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:07,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:07,503 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:07,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:07,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:07,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:07,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:07,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:07,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:07,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:07,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:08,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:08,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:08,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:08,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,295 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:08,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:08,339 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:08,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:08,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:08,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:08,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:08,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:08,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:08,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:08,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:08,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:08,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:09,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:09,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,089 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:09,160 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:09,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:09,301 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:09,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,413 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:10:09,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:10:09,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:10:09,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:10:09,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,704 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:10:09,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:10:09,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,841 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:10:09,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:10:09,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:09,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:09,930 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:10:09,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:10:09,964 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:10:09,977 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:10:09,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:10:09,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:10,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:10:10,019 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:10:10,019 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:10:10,023 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:10:10,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:10,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:10:10,558 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:10,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:10:10,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:10:10,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:10:10,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:10:10,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:10:10,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:10:10,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:10:10,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:10:10,694 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:10:10,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:10:10,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:10,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:10,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:10:10,834 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:10:10,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:10,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:10,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:10:10,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:10:10,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:10,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:10:11,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:10:11,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:10:11,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:10:11,147 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,190 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:10:11,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:10:11,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,240 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:11,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:10:11,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:11,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:10:11,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:11,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:10:11,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,426 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:11,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:10:11,471 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,507 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:11,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:10:11,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,572 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:11,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:10:11,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:11,645 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:10:11,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:11,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:10:11,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:11,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:10:11,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:11,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:10:11,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:11,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:10:11,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:11,966 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:11,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:11,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:10:12,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:12,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:12,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:12,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:10:12,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:12,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:12,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:12,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:10:12,150 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:12,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:10:12,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:12,198 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:10:12,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:10:12,221 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:10:12,221 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:10:12,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:10:12,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:10:12,256 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:10:12,256 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:10:12,257 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:10:12,262 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:10:12,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:10:12,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:10:12,674 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:10:12,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:10:12,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:10:12,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:10:12,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:10:12,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:10:12,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:10:12,793 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:10:12,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:13,003 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:13,009 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:13,009 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:15,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:15,180 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:15,186 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:15,187 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:17,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:17,331 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:17,335 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:17,335 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:19,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:19,538 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:19,542 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:19,542 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:21,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:21,697 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:21,701 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:21,701 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:23,701 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:23,848 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:23,851 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:23,852 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:25,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:26,017 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:26,022 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:26,023 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:28,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:28,180 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:28,184 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:28,184 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:30,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:30,368 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:30,371 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:30,372 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:32,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:32,555 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:32,559 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:32,559 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:34,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:34,736 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:34,740 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:34,740 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:36,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:36,868 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:36,872 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:36,872 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:38,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:39,055 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:10:39,059 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:39,059 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:41,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:41,217 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:41,217 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:43,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:43,365 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:43,365 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:45,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:45,522 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:45,522 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:47,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:47,674 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:47,674 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:49,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:49,822 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:49,822 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:51,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:51,969 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:51,969 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:53,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:54,125 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:54,125 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:56,127 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:56,278 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:56,278 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:10:58,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:10:58,480 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:10:58,480 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:00,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:00,644 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:00,644 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:02,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:02,803 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:02,803 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:04,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:04,993 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:04,993 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:06,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:07,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:07,154 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:09,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:09,309 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:09,309 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:11,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:11,469 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:11,469 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:13,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:13,631 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:13,632 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:15,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:15,796 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:15,796 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:17,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:17,947 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:17,947 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:19,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:20,104 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:20,105 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:22,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:22,265 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:22,265 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:24,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:24,426 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:24,426 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:26,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:26,582 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:26,582 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:28,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:28,747 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:28,749 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:30,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:30,908 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:30,908 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:32,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:33,075 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:33,075 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:35,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:35,223 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:35,223 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:37,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:37,387 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:37,388 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:39,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:39,552 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:39,552 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:41,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:41,744 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:41,744 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:43,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:44,114 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:44,115 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:46,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:46,345 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:46,345 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:48,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:48,528 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:48,529 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:50,529 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:50,713 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:11:50,713 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:11:52,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:53,469 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:11:53,469 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 101.211000204. 2021-12-16 15:11:53,470 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:11:53,473 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:11:53,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:11:53,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:11:54,009 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:11:54,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:11:54,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:11:54,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:11:54,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:11:54,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:11:54,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:11:54,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:11:54,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:54,371 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:11:54,371 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.901999950409. 2021-12-16 15:11:54,372 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:11:54,375 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:11:54,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:11:54,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:11:54,858 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:11:54,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:11:54,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:11:54,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:11:54,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:11:54,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:11:54,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:11:55,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:11:55,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:11:55,252 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:11:55,252 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.87900018692. ############################################################ # # RAM FootPrint: 3863.41015625 # ############################################################ 2021-12-16 15:11:55,283 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:11:55,283 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 75 seconds. Reason: Iteration:17 waiting to kill memc on all nodes 2021-12-16 15:13:10,289 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:13:10,289 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:13:15,292 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:13:15,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:16,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:13:16,673 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:16,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:13:16,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:13:16,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:13:16,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:13:16,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:13:16,821 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:13:16,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:13:16,855 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:13:16,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:13:16,938 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['29073 memcached\n'], []) 2021-12-16 15:13:16,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:13:17,010 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:13:17,150 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['29073 memcached\n'], []) 2021-12-16 15:13:17,150 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:13:17,160 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:13:17,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:17,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:13:17,670 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:17,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:13:17,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:13:17,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:13:17,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:13:17,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:13:17,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:13:17,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:13:17,796 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:13:17,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:13:17,872 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['219988 memcached\n'], []) 2021-12-16 15:13:17,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:13:17,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:13:18,022 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['219988 memcached\n'], []) 2021-12-16 15:13:18,023 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:13:18,026 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:13:18,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:18,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:13:18,512 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:18,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:13:18,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:13:18,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:13:18,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:13:18,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:13:18,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:13:18,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:13:18,684 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:13:18,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:13:18,770 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['63440 memcached\n'], []) 2021-12-16 15:13:18,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:13:18,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:13:18,911 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['63440 memcached\n'], []) 2021-12-16 15:13:18,913 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:13:18,915 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:13:19,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:19,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:13:19,312 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:19,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:13:19,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:13:19,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:13:19,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:13:19,395 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:13:19,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:13:19,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:13:19,444 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:13:19,461 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:13:19,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:13:19,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:19,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:19,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:19,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:19,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:19,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:19,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:19,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:19,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:19,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:19,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:19,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:19,950 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:19,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:20,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:20,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,122 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:20,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:20,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:20,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:20,249 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,270 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:20,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:20,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:20,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:20,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:20,502 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:20,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:20,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:20,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,660 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:20,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:20,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:20,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:20,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:20,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:20,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:20,920 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:20,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:20,966 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:20,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:13:21,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:13:21,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:13:21,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:13:21,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:13:21,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:13:21,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:13:21,267 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:13:21,283 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:13:21,358 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:13:21,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:21,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:13:21,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:13:21,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:13:21,460 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:13:21,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:13:21,474 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:21,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:13:21,505 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:13:21,505 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:13:21,507 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:13:21,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:24,144 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:13:24,226 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:24,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:13:24,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:13:24,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:13:24,285 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:13:24,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:13:24,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:13:24,335 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:13:24,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:13:24,368 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:13:24,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:13:24,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:13:24,505 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:13:24,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,566 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:24,584 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:24,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:24,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:24,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,704 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:24,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:24,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,752 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:24,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:24,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:24,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:24,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:24,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:24,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:24,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:24,999 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:25,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:25,035 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:25,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:25,108 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:25,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:25,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:25,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:25,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:25,328 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:25,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,371 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:25,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:25,436 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:25,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:25,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:25,588 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:25,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:13:25,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:13:25,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:13:25,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:13:25,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:13:25,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:13:25,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,901 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:13:25,918 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:13:25,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:25,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:25,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:13:25,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:13:26,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:13:26,013 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:13:26,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:13:26,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:26,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:13:26,053 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:13:26,053 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:13:26,056 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:13:26,184 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:26,499 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:13:26,581 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:26,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:13:26,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:13:26,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:13:26,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:13:26,644 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:13:26,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:13:26,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:13:26,684 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:13:26,697 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:13:26,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:13:26,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:26,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:26,813 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:13:26,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:13:26,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:26,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:26,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:13:26,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:13:26,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:26,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:26,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:13:27,006 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:13:27,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,053 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:13:27,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:13:27,107 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:13:27,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:13:27,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,219 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:13:27,233 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:13:27,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:27,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:13:27,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,345 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,374 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:27,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:13:27,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,443 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:27,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:13:27,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:27,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:13:27,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:27,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:13:27,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:27,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:13:27,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:27,849 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:13:27,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:27,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:27,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:13:27,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:27,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:28,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:13:28,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:28,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:13:28,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,216 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,259 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:28,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:13:28,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:28,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:13:28,450 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:28,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:13:28,578 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:13:28,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:28,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:13:28,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:13:28,714 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:13:28,716 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:13:28,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:13:28,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:13:28,769 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:13:28,769 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:13:28,769 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:13:28,772 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:13:28,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:13:29,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:13:29,273 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:13:29,273 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:13:29,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:13:29,318 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:13:29,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:13:29,368 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:13:29,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:13:29,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:13:29,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:29,602 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:29,608 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:29,608 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:31,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:31,788 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:31,792 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:31,792 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:33,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:33,963 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:33,967 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:33,967 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:35,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:36,144 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:36,150 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:36,150 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:38,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:38,316 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:38,321 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:38,321 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:40,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:40,493 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:40,497 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:40,497 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:42,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:42,681 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:42,686 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:42,686 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:44,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:44,849 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:44,854 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:44,854 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:46,854 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:46,954 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:46,959 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:46,959 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:48,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:49,130 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:49,134 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:49,134 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:51,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:51,305 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:51,309 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:51,309 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:53,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:53,474 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:13:53,479 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:53,479 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:55,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:55,628 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:55,628 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:57,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:57,792 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:57,792 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:13:59,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:13:59,970 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:13:59,970 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:01,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:02,122 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:02,124 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:04,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:04,273 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:04,273 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:06,275 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:06,424 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:06,424 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:08,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:08,575 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:08,575 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:10,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:10,733 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:10,733 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:12,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:12,888 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:12,888 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:14,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:15,052 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:15,052 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:17,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:17,230 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:17,230 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:19,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:19,395 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:19,395 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:21,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:21,552 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:21,552 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:23,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:23,717 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:23,717 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:25,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:25,878 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:25,878 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:27,878 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:28,055 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:28,055 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:30,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:30,217 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:30,217 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:32,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:32,375 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:32,375 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:34,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:34,525 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:34,526 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:36,526 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:36,680 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:36,680 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:38,680 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:38,838 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:38,839 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:40,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:40,999 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:41,000 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:43,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:43,161 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:43,161 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:45,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:45,332 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:45,332 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:47,332 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:47,493 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:47,493 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:49,494 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:49,651 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:49,651 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:51,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:52,042 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:52,042 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:54,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:54,611 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:54,611 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:56,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:56,855 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:56,855 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:14:58,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:14:59,035 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:14:59,036 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:15:01,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:15:01,835 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:15:01,835 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:15:03,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:15:04,128 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:15:04,128 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:15:06,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:15:06,299 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:15:06,301 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 97.5320000648. 2021-12-16 15:15:06,302 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:15:06,303 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:15:06,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:15:06,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:15:06,822 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:15:06,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:15:06,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:15:06,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:15:06,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:15:06,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:15:06,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:15:06,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:15:06,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:15:07,101 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:15:07,101 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.799000024796. 2021-12-16 15:15:07,101 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:15:07,105 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:15:07,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:15:07,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:15:07,562 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:15:07,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:15:07,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:15:07,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:15:07,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:15:07,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:15:07,676 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:15:07,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:15:07,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:15:07,898 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:15:07,898 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.796999931335. ############################################################ # # RAM FootPrint: 3804.296875 # ############################################################ 2021-12-16 15:15:07,928 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:15:07,930 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 94 seconds. Reason: Iteration:18 waiting to kill memc on all nodes 2021-12-16 15:16:41,933 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:16:41,933 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:16:46,934 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:16:47,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:47,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:16:47,953 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:47,954 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:16:47,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:16:47,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:16:48,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:16:48,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:16:48,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:16:48,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:16:48,095 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:16:48,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:16:48,160 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['30943 memcached\n'], []) 2021-12-16 15:16:48,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:16:48,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:16:48,321 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['30943 memcached\n'], []) 2021-12-16 15:16:48,321 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:16:48,325 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:16:48,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:48,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:16:48,845 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:48,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:16:48,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:16:48,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:16:48,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:16:48,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:16:48,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:16:48,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:16:48,977 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:16:48,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:16:49,030 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['220972 memcached\n'], []) 2021-12-16 15:16:49,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:16:49,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:16:49,163 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['220972 memcached\n'], []) 2021-12-16 15:16:49,164 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:16:49,167 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:16:49,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:49,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:16:49,635 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:49,637 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:16:49,660 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:16:49,683 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:16:49,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:16:49,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:16:49,765 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:16:49,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:16:49,812 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:16:49,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:16:49,897 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['65003 memcached\n'], []) 2021-12-16 15:16:49,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:16:49,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:16:50,039 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['65003 memcached\n'], []) 2021-12-16 15:16:50,039 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:16:50,042 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:16:50,151 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:50,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:16:50,476 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:50,476 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:16:50,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:16:50,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:16:50,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:16:50,553 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:16:50,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:16:50,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:16:50,602 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:16:50,618 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:16:50,647 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:16:50,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:50,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:50,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:50,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:50,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:50,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:50,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:50,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:50,892 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:50,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:50,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:50,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:51,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:51,115 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:51,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,211 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:51,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:51,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,299 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,341 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:51,369 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:51,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:51,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:51,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:51,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:51,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:51,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:51,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:51,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:51,826 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:51,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:51,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:51,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:51,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:52,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:52,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:52,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:52,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,253 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:52,283 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:52,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:52,424 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:52,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,489 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,530 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:16:52,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:16:52,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:16:52,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:16:52,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:16:52,780 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:16:52,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:16:52,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:16:52,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:52,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:16:52,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:16:52,956 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:16:52,969 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:16:52,969 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:16:52,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:52,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:16:53,009 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:16:53,009 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:16:53,012 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:16:53,124 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:53,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:16:53,505 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:53,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:16:53,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:16:53,542 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:16:53,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:16:53,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:16:53,596 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:16:53,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:16:53,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:16:53,642 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:16:53,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:16:53,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:53,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:53,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:16:53,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:16:53,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:53,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:53,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:53,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:53,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:53,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:53,993 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:54,016 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:54,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:54,132 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:54,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:54,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:54,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:54,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:54,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:54,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:54,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:54,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:54,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,760 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,801 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:54,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:54,858 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:54,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:54,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:54,961 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:54,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,033 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,063 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:55,095 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:55,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:55,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:55,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,354 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:55,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:55,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:55,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:55,549 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,586 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:55,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:55,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:55,755 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:55,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:16:55,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:16:55,852 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:16:55,923 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:16:55,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:55,973 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:55,996 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:16:56,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:16:56,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:56,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:56,079 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:16:56,096 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:16:56,119 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:16:56,138 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:16:56,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:16:56,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:56,178 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:16:56,194 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:16:56,194 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:16:56,197 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:16:56,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:56,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:16:56,707 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:56,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:16:56,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:16:56,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:16:56,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:16:56,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:16:56,786 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:16:56,799 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:16:56,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:16:56,832 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:16:56,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:16:56,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:56,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:56,948 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:16:56,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:16:56,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:16:57,098 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:16:57,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,196 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:16:57,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:16:57,255 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:16:57,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:16:57,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,456 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:16:57,479 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:16:57,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,565 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:57,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:16:57,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,658 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:57,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:16:57,736 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:57,815 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:16:57,835 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,859 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,885 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:57,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:16:57,926 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:57,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:57,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:57,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:16:58,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:58,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:16:58,109 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,132 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:58,206 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:16:58,232 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:58,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:16:58,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,367 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:58,421 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:16:58,446 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,470 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:58,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:16:58,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:58,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:16:58,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:58,719 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:16:58,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,772 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:58,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:16:58,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:58,933 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:58,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:16:58,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:58,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:16:59,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:59,030 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:16:59,043 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:16:59,056 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:16:59,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:16:59,069 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:16:59,091 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:16:59,108 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:16:59,108 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:16:59,108 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:16:59,114 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:16:59,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:16:59,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:16:59,650 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:16:59,651 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:16:59,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:16:59,691 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:16:59,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:16:59,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:16:59,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:16:59,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:16:59,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:16:59,980 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:16:59,987 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:16:59,987 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:01,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:02,167 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:02,173 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:02,173 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:04,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:04,359 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:04,365 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:04,365 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:06,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:06,530 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:06,535 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:06,535 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:08,536 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:08,703 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:08,707 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:08,709 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:10,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:10,875 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:10,880 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:10,881 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:12,881 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:13,049 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:13,053 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:13,053 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:15,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:15,174 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:15,177 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:15,177 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:17,177 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:17,345 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:17,351 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:17,351 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:19,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:19,523 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:19,528 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:19,528 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:21,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:21,694 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:21,697 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:21,697 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:23,698 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:23,887 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:23,891 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:23,891 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:25,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:26,065 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:26,071 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:26,071 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:28,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:28,239 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:28,243 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:28,243 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:30,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:30,411 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:30,414 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:30,414 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:32,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:32,578 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:17:32,584 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:32,584 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:34,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:34,737 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:34,737 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:36,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:36,888 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:36,890 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:38,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:39,036 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:39,036 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:41,036 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:41,198 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:41,200 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:43,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:43,361 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:43,361 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:45,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:45,523 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:45,523 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:47,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:47,668 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:47,668 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:49,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:49,819 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:49,819 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:51,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:51,986 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:51,986 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:53,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:54,154 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:54,154 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:56,155 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:56,321 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:56,321 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:17:58,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:17:58,480 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:17:58,480 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:00,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:00,642 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:00,642 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:02,642 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:02,808 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:02,808 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:04,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:04,977 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:04,979 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:06,979 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:07,148 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:07,148 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:09,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:09,309 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:09,309 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:11,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:11,471 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:11,473 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:13,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:13,631 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:13,631 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:15,632 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:15,793 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:15,795 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:17,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:17,956 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:17,957 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:19,957 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:20,121 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:20,121 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:22,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:22,276 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:22,278 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:24,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:24,512 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:24,512 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:26,513 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:26,696 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:26,696 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:28,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:28,996 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:28,996 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:30,997 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:31,693 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:31,693 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:33,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:34,000 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:34,002 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:36,002 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:36,167 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:36,167 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:38,168 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:38,414 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:38,414 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:40,414 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:41,181 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:41,183 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:43,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:43,627 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:43,627 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:45,627 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:45,792 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:18:45,792 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:18:47,792 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:48,384 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:18:48,384 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.276000023. 2021-12-16 15:18:48,385 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:18:48,388 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:18:48,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:18:48,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:18:48,951 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:18:48,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:18:48,971 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:18:48,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:18:49,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:18:49,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:18:49,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:18:49,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:18:49,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:49,260 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:18:49,260 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.876999855042. 2021-12-16 15:18:49,263 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:18:49,265 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:18:49,407 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:18:49,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:18:49,757 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:18:49,757 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:18:49,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:18:49,808 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:18:49,836 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:18:49,864 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:18:49,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:18:49,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:18:49,937 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:18:50,158 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:18:50,160 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.897000074387. ############################################################ # # RAM FootPrint: 3735.1640625 # ############################################################ 2021-12-16 15:18:50,197 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:18:50,197 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 100 seconds. Reason: Iteration:19 waiting to kill memc on all nodes 2021-12-16 15:20:30,203 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:20:30,203 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:20:35,206 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:20:35,336 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:35,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:20:35,798 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:35,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:20:35,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:20:35,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:20:35,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:20:35,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:20:35,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:20:35,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:20:35,903 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:20:35,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:20:35,966 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['32802 memcached\n'], []) 2021-12-16 15:20:35,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:20:36,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:20:36,055 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['32802 memcached\n'], []) 2021-12-16 15:20:36,055 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:20:36,059 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:20:36,141 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:36,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:20:36,450 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:36,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:20:36,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:20:36,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:20:36,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:20:36,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:20:36,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:20:36,535 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:20:36,548 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:20:36,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:20:36,598 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['222559 memcached\n'], []) 2021-12-16 15:20:36,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:20:36,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:20:36,723 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['222559 memcached\n'], []) 2021-12-16 15:20:36,723 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:20:36,726 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:20:36,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:37,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:20:37,165 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:37,165 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:20:37,191 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:20:37,213 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:20:37,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:20:37,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:20:37,279 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:20:37,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:20:37,313 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:20:37,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:20:37,390 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['66573 memcached\n'], []) 2021-12-16 15:20:37,391 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:20:37,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:20:37,542 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['66573 memcached\n'], []) 2021-12-16 15:20:37,542 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:20:37,545 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:20:37,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:37,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:20:37,993 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:37,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:20:38,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:20:38,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:20:38,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:20:38,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:20:38,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:20:38,104 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:20:38,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:20:38,138 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:20:38,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:20:38,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,224 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,262 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:38,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:38,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,349 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:38,408 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:38,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,467 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,500 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:38,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:38,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:38,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:38,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:38,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:38,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,796 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,821 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:38,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:38,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:38,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:38,938 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:38,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:38,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:39,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:39,025 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,066 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:39,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:39,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,121 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:39,176 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:39,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:39,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:39,280 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:39,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:39,361 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,431 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:39,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:39,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:39,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:39,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,638 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:39,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:39,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,769 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:20:39,790 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:20:39,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,874 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:20:39,894 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:20:39,913 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:39,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:39,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:20:39,987 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:20:40,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:40,042 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:40,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:20:40,112 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:20:40,142 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:40,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:40,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:20:40,223 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:20:40,247 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:20:40,270 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:20:40,270 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:20:40,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:40,308 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:20:40,322 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:20:40,322 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:20:40,325 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:20:40,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:40,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:20:40,822 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:40,822 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:20:40,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:20:40,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:20:40,868 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:20:40,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:20:40,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:20:40,910 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:20:40,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:20:40,937 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:20:40,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:20:40,979 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,009 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,040 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:20:41,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:20:41,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,114 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:41,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:41,174 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:41,246 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:41,269 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:41,329 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:41,346 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,428 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:41,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:41,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,539 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:41,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:41,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,605 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,631 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:41,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:41,668 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,746 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:41,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:41,803 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:41,882 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:41,914 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:41,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:41,974 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,015 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:42,046 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:42,071 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,138 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:42,170 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:42,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:42,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:42,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:42,433 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:42,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,502 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,533 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:42,562 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:42,585 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:42,707 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:42,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,839 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:42,872 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:42,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:42,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:42,977 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:20:43,003 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:20:43,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:43,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:43,086 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:20:43,105 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:20:43,128 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:43,161 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:43,200 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:20:43,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:20:43,252 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:43,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:43,313 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:20:43,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:20:43,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:20:43,382 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:20:43,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:20:43,398 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:43,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:20:43,434 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:20:43,434 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:20:43,437 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:20:43,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:43,884 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:20:43,960 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:43,960 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:20:43,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:20:43,990 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:20:44,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:20:44,026 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:20:44,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:20:44,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:20:44,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:20:44,078 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:20:44,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:20:44,125 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,157 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:20:44,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:20:44,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:20:44,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:20:44,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,364 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:20:44,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:20:44,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,440 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:20:44,457 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:20:44,473 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,490 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:20:44,523 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:20:44,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,555 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:44,589 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:20:44,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,621 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:44,653 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:20:44,667 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:44,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:20:44,734 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,789 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:44,811 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:20:44,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:44,905 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:44,934 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:20:44,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:44,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,012 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:45,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:20:45,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,075 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,099 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:45,118 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:20:45,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,164 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:45,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:20:45,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,276 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:45,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:20:45,325 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,357 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:45,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:20:45,433 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,464 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:45,522 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:20:45,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,571 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,594 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:45,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:20:45,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,670 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:45,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:20:45,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,721 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,742 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:45,756 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:20:45,770 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:20:45,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:45,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:20:45,821 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:20:45,832 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:20:45,832 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:20:45,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:20:45,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:20:45,865 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:20:45,865 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:20:45,865 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:20:45,868 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:20:45,992 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:20:46,303 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:20:46,377 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:20:46,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:20:46,397 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:20:46,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:20:46,438 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:20:46,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:20:46,480 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:20:46,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:20:46,517 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:46,690 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:46,696 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:46,697 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:48,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:48,868 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:48,874 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:48,875 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:50,875 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:51,046 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:51,052 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:51,052 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:53,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:53,232 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:53,236 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:53,236 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:55,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:55,414 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:55,418 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:55,418 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:57,420 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:57,589 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:57,595 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:57,595 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:20:59,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:20:59,779 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:20:59,783 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:20:59,783 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:01,785 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:01,959 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:01,963 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:01,963 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:03,963 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:04,131 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:04,135 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:04,135 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:06,137 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:06,315 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:06,319 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:06,319 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:08,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:08,505 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:08,509 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:08,509 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:10,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:10,670 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:10,674 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:10,674 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:12,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:12,859 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:12,864 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:12,864 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:14,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:15,026 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:15,030 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:15,030 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:17,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:17,190 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:17,194 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:17,194 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:19,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:19,359 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:21:19,365 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:19,365 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:21,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:21,528 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:21,528 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:23,528 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:23,737 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:23,739 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:25,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:25,898 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:25,898 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:27,900 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:28,061 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:28,061 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:30,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:30,230 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:30,230 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:32,230 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:32,387 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:32,387 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:34,387 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:34,578 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:34,578 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:36,579 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:36,729 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:36,730 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:38,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:38,934 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:38,936 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:40,936 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:41,102 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:41,102 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:43,102 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:43,276 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:43,276 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:45,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:45,447 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:45,447 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:47,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:47,599 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:47,601 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:49,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:49,752 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:49,752 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:51,753 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:51,917 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:51,917 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:53,917 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:54,079 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:54,079 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:56,081 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:56,234 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:56,236 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:21:58,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:21:58,398 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:21:58,400 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:00,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:00,559 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:00,561 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:02,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:02,727 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:02,729 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:04,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:04,907 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:04,907 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:06,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:07,066 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:07,068 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:09,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:09,226 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:09,226 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:11,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:11,382 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:11,382 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:13,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:13,568 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:13,568 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:15,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:15,730 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:15,730 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:17,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:17,885 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:17,885 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:19,887 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:20,049 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:20,051 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:22,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:22,224 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:22,224 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:24,224 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:24,378 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:24,378 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:26,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:26,561 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:26,561 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:28,561 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:28,727 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:28,727 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:30,729 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:30,894 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:30,895 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:32,895 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:33,056 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:22:33,056 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:22:35,058 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:35,260 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:22:35,260 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 109.394999981. 2021-12-16 15:22:35,262 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:22:35,263 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:22:35,384 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:22:35,714 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:22:35,795 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:22:35,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:22:35,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:22:35,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:22:35,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:22:35,865 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:22:35,904 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:22:35,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:22:35,944 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:36,161 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:22:36,161 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.899999856949. 2021-12-16 15:22:36,161 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:22:36,164 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:22:36,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:22:36,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:22:36,640 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:22:36,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:22:36,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:22:36,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:22:36,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:22:36,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:22:36,762 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:22:36,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:22:36,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:22:37,006 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:22:37,006 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.84500002861. ############################################################ # # RAM FootPrint: 3640.48046875 # ############################################################ 2021-12-16 15:22:37,039 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L827 2021-12-16 15:22:37,039 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 67 seconds. Reason: Iteration:20 waiting to kill memc on all nodes 2021-12-16 15:23:44,043 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> kill_memcached():L843 2021-12-16 15:23:44,045 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Sleep for 5 seconds between continuous memc kill 2021-12-16 15:23:49,046 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:23:49,510 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:50,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:23:50,568 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:50,568 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:23:50,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:23:50,595 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:23:50,612 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:23:50,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:23:50,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:23:50,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:23:50,729 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.65 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:23:50,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:23:50,812 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.65 - Process info before sending signal: (['34728 memcached\n'], []) 2021-12-16 15:23:50,812 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: kill -9 $(pgrep memcached) 2021-12-16 15:23:50,903 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: pgrep -l memcached 2021-12-16 15:23:51,063 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.65 - Process info after sending signal: (['34728 memcached\n'], []) 2021-12-16 15:23:51,065 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:23:51,068 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:23:51,187 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:51,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:23:51,602 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:51,604 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:23:51,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:23:51,648 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:23:51,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:23:51,693 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:23:51,709 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:23:51,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:23:51,740 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.66 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:23:51,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:23:51,802 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.66 - Process info before sending signal: (['224021 memcached\n'], []) 2021-12-16 15:23:51,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: kill -9 $(pgrep memcached) 2021-12-16 15:23:51,851 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: pgrep -l memcached 2021-12-16 15:23:51,924 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.66 - Process info after sending signal: (['224021 memcached\n'], []) 2021-12-16 15:23:51,924 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:23:51,927 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:23:52,032 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:52,288 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:23:52,377 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:52,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:23:52,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:23:52,418 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:23:52,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:23:52,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:23:52,482 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:23:52,497 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:23:52,516 | infra | DEBUG | Thread-5 | [remote_util:kill_process:534] 172.23.110.64 - Sending 9 signal to process: memcached, service: memcached 2021-12-16 15:23:52,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:23:52,576 | infra | DEBUG | Thread-5 | [remote_util:kill_process:541] 172.23.110.64 - Process info before sending signal: (['68200 memcached\n'], []) 2021-12-16 15:23:52,576 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: kill -9 $(pgrep memcached) 2021-12-16 15:23:52,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: pgrep -l memcached 2021-12-16 15:23:52,733 | infra | DEBUG | Thread-5 | [remote_util:kill_process:546] 172.23.110.64 - Process info after sending signal: (['68200 memcached\n'], []) 2021-12-16 15:23:52,733 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:23:52,736 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:23:52,857 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:53,056 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:23:53,134 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:53,134 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:23:53,153 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:23:53,171 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:23:53,188 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:23:53,207 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:23:53,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:23:53,234 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:23:53,250 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:23:53,266 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:23:53,290 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:23:53,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,378 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:53,405 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:53,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,487 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:53,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:53,532 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:53,615 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:53,641 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,664 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,687 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:53,706 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:53,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,743 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,766 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:53,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:53,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:53,855 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:53,869 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:53,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:53,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:53,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:53,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:54,005 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:54,020 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,061 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:54,076 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:54,092 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,111 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:54,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:54,163 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,183 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,203 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:54,220 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:54,243 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:54,302 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:54,319 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,342 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:54,382 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:54,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,423 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,447 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:23:54,466 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:23:54,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,509 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,538 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:23:54,556 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:23:54,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,609 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:23:54,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:23:54,671 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,696 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:23:54,740 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:23:54,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,782 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:23:54,825 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:23:54,844 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,867 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,890 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:23:54,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:23:54,927 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:54,947 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:54,967 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:23:54,984 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:23:55,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:23:55,023 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:23:55,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:23:55,038 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:55,055 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:23:55,071 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:23:55,071 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:23:55,073 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:23:55,201 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:55,519 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:23:55,608 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:55,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:23:55,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:23:55,645 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:23:55,665 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:23:55,686 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:23:55,703 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:23:55,717 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:23:55,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:23:55,750 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:23:55,776 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:23:55,798 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:55,838 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:55,880 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:23:55,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:23:55,940 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:55,980 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:56,048 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:56,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,107 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,140 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:56,167 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:56,193 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,226 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,266 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:56,289 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:56,306 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,348 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:56,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:56,381 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,453 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:56,484 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:56,512 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,552 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,582 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:56,608 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:56,628 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:56,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:56,720 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,750 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,773 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:56,788 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:56,802 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,831 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,861 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:56,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:56,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:56,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:56,953 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:56,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:56,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,007 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,029 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:57,045 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:57,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,101 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:57,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:57,131 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,158 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,180 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:57,194 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:57,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:23:57,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:23:57,292 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,312 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,338 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:23:57,355 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:23:57,372 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,392 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:23:57,427 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:23:57,441 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,460 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,477 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:23:57,492 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:23:57,506 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,525 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,543 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:23:57,559 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:23:57,573 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,592 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:57,611 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:23:57,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:23:57,640 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:23:57,650 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:23:57,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:23:57,661 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:57,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:23:57,686 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:23:57,686 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:23:57,691 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:23:57,805 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:23:58,130 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:23:58,207 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:23:58,209 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:23:58,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:23:58,244 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:23:58,263 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:23:58,282 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:23:58,296 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:23:58,311 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:23:58,326 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:23:58,341 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:23:58,377 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:23:58,400 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,434 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:23:58,493 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:23:58,515 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,548 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,575 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:23:58,601 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:23:58,625 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,678 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:23:58,697 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:23:58,713 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,739 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,763 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:23:58,779 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:23:58,795 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,818 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,842 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:23:58,857 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:23:58,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,888 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,907 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:58,921 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:23:58,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:58,959 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:58,976 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:58,989 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:23:59,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,019 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:59,052 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:23:59,065 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,082 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,117 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:59,135 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:23:59,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,186 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,210 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:59,237 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:23:59,260 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,283 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,315 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:59,334 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:23:59,351 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,375 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,401 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:59,415 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:23:59,430 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,448 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,469 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:59,483 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:23:59,496 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,516 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,546 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:59,563 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:23:59,581 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,617 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,654 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:59,673 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:23:59,690 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,730 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:59,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:23:59,759 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,778 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,809 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:59,828 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:23:59,846 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,877 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:59,915 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:23:59,931 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:23:59,951 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:23:59,970 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:23:59,986 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:24:00,000 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:24:00,013 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:24:00,028 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:24:00,039 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:24:00,051 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:24:00,062 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:24:00,062 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:24:00,072 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:24:00,085 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:24:00,095 | test | DEBUG | Thread-5 | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:24:00,095 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:24:00,095 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:24:00,099 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:24:00,217 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:24:00,520 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:24:00,598 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:24:00,598 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:24:00,618 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:24:00,635 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:24:00,657 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:24:00,677 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:24:00,694 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:24:00,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:24:00,726 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:00,841 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: 'ep_warmup_thread' 2021-12-16 15:24:00,845 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:00,845 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:02,845 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:03,016 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:03,023 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:03,023 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:05,023 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:05,197 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:05,200 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:05,200 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:07,204 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:07,358 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:07,361 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:07,362 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:09,362 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:09,539 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:09,543 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:09,545 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:11,545 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:11,719 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:11,723 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:11,723 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:13,723 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:13,894 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:13,898 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:13,898 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:15,898 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:16,084 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:16,088 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:16,088 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:18,088 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:18,273 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:18,278 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:18,278 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:20,278 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:20,447 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:20,451 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:20,451 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:22,451 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:22,631 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:22,634 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:22,634 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:24,634 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:24,813 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:24,818 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:24,818 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:26,819 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:26,990 | test | WARNING | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4190] Exception during cbstat all cmd: Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1012, in main() File "/opt/couchbase/lib/python/cbstats", line 1009, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 54, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-12-16 15:24:26,994 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:26,994 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:28,994 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:29,147 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:29,147 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:31,148 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:31,309 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:31,309 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:33,309 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:33,463 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:33,463 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:35,463 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:35,614 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:35,614 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:37,614 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:37,766 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:37,766 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:39,767 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:39,923 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:39,924 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:41,924 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:42,066 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:42,066 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:44,068 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:44,256 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:44,257 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:46,257 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:46,417 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:46,417 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:48,417 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:48,579 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:48,579 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:50,579 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:50,743 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:50,743 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:52,744 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:52,911 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:52,911 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:54,911 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:55,073 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:55,073 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:57,073 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:57,236 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:57,236 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:24:59,236 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:24:59,403 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:24:59,403 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:01,403 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:01,571 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:01,571 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:03,571 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:03,733 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:03,733 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:05,733 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:05,895 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:05,897 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:07,897 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:08,059 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:08,059 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:10,059 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:10,227 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:10,227 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:12,227 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:12,388 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:12,388 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:14,388 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:14,552 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:14,552 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:16,553 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:16,709 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:16,709 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:18,710 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:18,871 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:18,871 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:20,871 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:21,539 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:21,539 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:23,540 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:23,747 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:23,749 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:25,749 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:25,927 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:25,927 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:27,928 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:28,105 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:28,107 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:30,107 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:30,285 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:30,286 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:32,286 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:32,457 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:32,459 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:34,459 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:34,622 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:34,624 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:36,624 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:36,890 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> _wait_warmup_completed():L4191 2021-12-16 15:25:36,891 | test | INFO | Thread-5 | [common_lib:sleep:22] Sleep 2 seconds. Reason: Warm-up not complete for GleamBookUsers0 on 172.23.110.65 2021-12-16 15:25:38,891 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:39,043 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:25:39,045 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 98.9490001202. 2021-12-16 15:25:39,045 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:25:39,046 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:25:39,154 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:25:39,486 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:25:39,569 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:25:39,569 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:25:39,589 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:25:39,607 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:25:39,630 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:25:39,650 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:25:39,663 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:25:39,674 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:25:39,688 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:39,832 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:25:39,834 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.789000034332. 2021-12-16 15:25:39,834 | test | DEBUG | Thread-5 | [bucket_ready_functions:_wait_warmup_completed:4174] Waiting for bucket GleamBookUsers0 to complete warm-up 2021-12-16 15:25:39,835 | infra | DEBUG | Thread-5 | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:25:39,941 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:25:40,197 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:25:40,283 | infra | DEBUG | Thread-5 | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:25:40,285 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:25:40,305 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:25:40,322 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:25:40,344 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:25:40,365 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:25:40,380 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:25:40,394 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:25:40,411 | infra | DEBUG | Thread-5 | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b GleamBookUsers0 all 2021-12-16 15:25:40,558 | infra | DEBUG | Thread-5 | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:25:40,559 | test | INFO | Thread-5 | [opd:check_warmup_complete:881] Bucket:GleamBookUsers0 warm-up completed in 0.724999904633. 2021-12-16 15:25:40,562 | test | DEBUG | Thread-5 | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/opd.py -> crash_memcached():L835 2021-12-16 15:25:40,562 | test | DEBUG | Thread-5 | [common_lib:sleep:24] Sleep 300 seconds. Reason: None ############################################################ # # Step 12: Failover a node and FullRecovery that node # ############################################################ 2021-12-16 15:30:40,717 | test | DEBUG | MainThread | [rest_client:fail_over:1374] ns_1@172.23.110.66 - Failover successful 2021-12-16 15:30:40,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/aGoodDoctor/Hospital.py -> SystemTestMagma():L617 2021-12-16 15:30:40,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 30 seconds. Reason: None 2021-12-16 15:31:10,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1660 2021-12-16 15:31:10,747 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-12-16 15:31:15,769 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1575 2021-12-16 15:31:15,769 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-12-16 15:31:18,780 | test | INFO | MainThread | [rest_client:monitorRebalance:1584] Rebalance done. Taken 8.0569999218 seconds to complete 2021-12-16 15:31:18,786 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1587 2021-12-16 15:31:18,786 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 8.0569999218 seconds. Reason: Wait after rebalance complete 2021-12-16 15:31:26,842 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1382] ns_1@172.23.110.66 - Setting recoveryType=full 2021-12-16 15:31:26,875 | test | DEBUG | MainThread | [rest_client:set_recovery_type:1397] ns_1@172.23.110.66 - RecoveryType set successful 2021-12-16 15:31:26,903 | infra | INFO | MainThread | [task_manager:add_new_task:22] Added new task: Rebalance_task_IN=[]_OUT=[]_1639697486.88 2021-12-16 15:31:26,904 | infra | DEBUG | MainThread | [task_manager:get_task_result:25] Getting task result for Rebalance_task_IN=[]_OUT=[]_1639697486.88 2021-12-16 15:31:26,904 | infra | INFO | pool-3-thread-8 | [task:start_task:90] Thread 'Rebalance_task_IN=[]_OUT=[]_1639697486.88' started 2021-12-16 15:31:26,933 | test | DEBUG | pool-3-thread-8 | [rest_client:rebalance:1439] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.110.64,ns_1@172.23.110.66,ns_1@172.23.110.65', 'ejectedNodes': '', 'user': 'Administrator'} 2021-12-16 15:31:26,984 | test | DEBUG | pool-3-thread-8 | [rest_client:rebalance:1444] Rebalance operation started 2021-12-16 15:31:26,987 | test | INFO | pool-3-thread-8 | [table_view:display:72] Rebalance Overview +---------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +---------------+----------+-----------------------+---------------+--------------+ | 172.23.110.64 | kv | 7.1.0-1913-enterprise | 11.9582720717 | Cluster node | | 172.23.110.66 | kv | 7.1.0-1913-enterprise | 10.0041841004 | Cluster node | | 172.23.110.65 | kv | 7.1.0-1913-enterprise | 12.3101686612 | Cluster node | +---------------+----------+-----------------------+---------------+--------------+ 2021-12-16 15:31:26,996 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:31:26,996 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:31:27,009 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:31:27,012 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:31:27,012 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:31:37,023 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:31:37,025 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:31:37,036 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:31:37,038 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:31:37,038 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:31:47,049 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:31:47,049 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:31:47,062 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:31:47,063 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:31:47,063 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:31:57,076 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:31:57,076 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:31:57,091 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:31:57,092 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:31:57,092 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:32:07,105 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:32:07,105 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:32:07,121 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:32:07,124 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:32:07,124 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:32:17,135 | test | DEBUG | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1656] Rebalance percentage: 0.00 % 2021-12-16 15:32:17,135 | test | INFO | pool-3-thread-8 | [task:check:385] Rebalance - status: running, progress: 0.0 2021-12-16 15:32:17,153 | infra | DEBUG | pool-3-thread-8 | [task:check:412] Wait before next rebalance progress check 2021-12-16 15:32:17,155 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L413 2021-12-16 15:32:17,155 | infra | DEBUG | pool-3-thread-8 | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-12-16 15:32:27,170 | test | ERROR | pool-3-thread-8 | [rest_client:_rebalance_status_and_progress:1651] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'27381b91f6e622a7892446345d0c5ff8', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=86700585e2d76da5960c84b825be0bd7', u'status': u'notRunning'} - rebalance failed 2021-12-16 15:32:27,200 | test | INFO | pool-3-thread-8 | [rest_client:print_UI_logs:2803] Latest logs from UI on 172.23.110.64: 2021-12-16 15:32:27,200 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697546942L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:32:26.942Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.110.66\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["GleamBookUsers0"]}}}]}.\nRebalance Operation Id = d5675b416af8bc4b53bfefc42faf8f24'} 2021-12-16 15:32:27,200 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697546940L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:32:26.940Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.110.66\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["GleamBookUsers0"]}}}]\n'} 2021-12-16 15:32:27,200 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697486935L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:26.935Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.110.64','ns_1@172.23.110.66',\n 'ns_1@172.23.110.65'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d5675b416af8bc4b53bfefc42faf8f24"} 2021-12-16 15:32:27,200 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697469581L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:09.581Z', u'text': u'Graceful failover completed successfully.\nRebalance Operation Id = eb0c25d0b2ac1b901ec0baede7f8d38f'} 2021-12-16 15:32:27,200 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.110.66', u'tstamp': 1639697469550L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:09.550Z', u'text': u'Shutting down bucket "GleamBookUsers0" on \'ns_1@172.23.110.66\' for deletion'} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697469462L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:09.462Z', u'text': u"Deactivating failed over nodes ['ns_1@172.23.110.66']"} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697469453L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:09.453Z', u'text': u"Failed over ['ns_1@172.23.110.66']: ok"} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'failover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697469272L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:31:09.272Z', u'text': u"Starting failing over ['ns_1@172.23.110.66']"} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697440746L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:30:40.746Z', u'text': u'Bucket "GleamBookUsers0" rebalance does not seem to be swap rebalance'} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [rest_client:print_UI_logs:2805] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.64', u'tstamp': 1639697440666L, u'shortText': u'message', u'serverTime': u'2021-12-16T15:30:40.666Z', u'text': u"Starting vbucket moves for graceful failover of ['ns_1@172.23.110.66']"} 2021-12-16 15:32:27,201 | test | ERROR | pool-3-thread-8 | [task:call:297] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'27381b91f6e622a7892446345d0c5ff8', u'subtype': u'rebalance', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=86700585e2d76da5960c84b825be0bd7', u'status': u'notRunning'} - rebalance failed FAIL 2021-12-16 15:32:27,203 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-16 15:32:27,210 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-16 15:32:27,210 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force 2021-12-16 15:32:27,213 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.64 with username: root, password: couchbase 2021-12-16 15:32:27,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:32:27,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /etc/ 2021-12-16 15:32:27,562 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:32:27,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /cygdrive/c/Windows 2021-12-16 15:32:27,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: uname -m 2021-12-16 15:32:27,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/cpuinfo 2021-12-16 15:32:27,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: df -Thl 2021-12-16 15:32:27,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: cat /proc/meminfo 2021-12-16 15:32:27,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname 2021-12-16 15:32:27,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: hostname -d 2021-12-16 15:32:27,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:32:27,678 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.64: No crash files found 2021-12-16 15:32:27,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:32:27,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:27,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:27,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:27,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:27,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:27,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:27,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:27,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:27,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:27,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:28,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:28,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,145 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:28,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:28,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:28,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:28,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:28,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:28,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:28,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:28,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:28,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:28,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:28,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:28,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:28,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:28,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:28,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:28,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:28,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:28,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:28,964 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:28,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,025 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:29,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:29,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:29,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:29,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,275 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:32:29,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:32:29,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:32:29,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:32:29,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:32:29,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:32:29,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:32:29,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:32:29,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:32:29,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:32:29,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:29,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt 2021-12-16 15:32:29,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000030.txt 2021-12-16 15:32:29,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:32:29,755 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'sanitizers.log.*' files found 2021-12-16 15:32:29,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:32:29,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:29,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.64: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:32:29,813 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.64: No 'indexer.log*' files found 2021-12-16 15:32:29,813 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.64 2021-12-16 15:32:29,818 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.65 with username: root, password: couchbase 2021-12-16 15:32:29,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:32:30,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /etc/ 2021-12-16 15:32:30,289 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:32:30,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /cygdrive/c/Windows 2021-12-16 15:32:30,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: uname -m 2021-12-16 15:32:30,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/cpuinfo 2021-12-16 15:32:30,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: df -Thl 2021-12-16 15:32:30,344 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: cat /proc/meminfo 2021-12-16 15:32:30,355 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname 2021-12-16 15:32:30,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: hostname -d 2021-12-16 15:32:30,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:32:30,404 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.65: No crash files found 2021-12-16 15:32:30,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:32:30,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:30,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:30,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:30,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:30,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:30,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:30,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:30,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:30,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:30,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:30,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:30,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:30,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:30,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:30,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:30,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:30,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:30,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:30,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:30,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:30,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:31,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:31,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:31,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:31,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:31,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:31,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:31,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:31,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:31,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:31,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,742 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:31,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:31,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:31,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:31,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:31,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:31,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:31,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:31,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:32,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:32,056 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,119 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:32,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:32,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:32:32,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000025.txt 2021-12-16 15:32:32,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:32:32,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000026.txt 2021-12-16 15:32:32,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:32:32,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000027.txt 2021-12-16 15:32:32,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:32:32,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000028.txt 2021-12-16 15:32:32,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:32,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:32:32,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000029.txt 2021-12-16 15:32:32,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:32:32,799 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'sanitizers.log.*' files found 2021-12-16 15:32:32,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:32:32,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:32,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.65: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:32:32,838 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.65: No 'indexer.log*' files found 2021-12-16 15:32:32,838 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.65 2021-12-16 15:32:32,839 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.66 with username: root, password: couchbase 2021-12-16 15:32:32,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:32:33,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /etc/ 2021-12-16 15:32:33,352 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:32:33,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /cygdrive/c/Windows 2021-12-16 15:32:33,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: uname -m 2021-12-16 15:32:33,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/cpuinfo 2021-12-16 15:32:33,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: df -Thl 2021-12-16 15:32:33,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: cat /proc/meminfo 2021-12-16 15:32:33,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname 2021-12-16 15:32:33,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: hostname -d 2021-12-16 15:32:33,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:32:33,476 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.66: No crash files found 2021-12-16 15:32:33,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:32:33,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:33,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:33,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:33,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:33,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:33,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:33,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:33,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:33,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:33,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:33,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:33,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:33,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:33,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:33,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:33,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:34,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:32:34,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:32:34,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:34,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:34,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:34,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000011.txt 2021-12-16 15:32:34,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:34,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000012.txt 2021-12-16 15:32:34,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:34,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-12-16 15:32:34,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:34,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000014.txt 2021-12-16 15:32:34,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,525 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:34,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000015.txt 2021-12-16 15:32:34,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:34,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000016.txt 2021-12-16 15:32:34,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:34,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000017.txt 2021-12-16 15:32:34,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:34,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-12-16 15:32:34,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:34,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000019.txt 2021-12-16 15:32:34,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,890 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:34,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000020.txt 2021-12-16 15:32:34,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:34,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:34,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:34,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000021.txt 2021-12-16 15:32:35,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:35,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:35,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:35,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000022.txt 2021-12-16 15:32:35,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:35,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:35,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:35,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000023.txt 2021-12-16 15:32:35,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:35,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:35,197 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:35,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000024.txt 2021-12-16 15:32:35,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:32:35,232 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'sanitizers.log.*' files found 2021-12-16 15:32:35,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:32:35,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:35,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.66: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:32:35,263 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.66: No 'indexer.log*' files found 2021-12-16 15:32:35,263 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.66 2021-12-16 15:32:35,269 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.67 with username: root, password: couchbase 2021-12-16 15:32:35,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:32:35,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /etc/ 2021-12-16 15:32:35,832 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:32:35,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /cygdrive/c/Windows 2021-12-16 15:32:35,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: uname -m 2021-12-16 15:32:35,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/cpuinfo 2021-12-16 15:32:35,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: df -Thl 2021-12-16 15:32:35,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: cat /proc/meminfo 2021-12-16 15:32:35,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname 2021-12-16 15:32:35,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: hostname -d 2021-12-16 15:32:35,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:32:35,973 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.67: No crash files found 2021-12-16 15:32:36,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:32:36,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 15:32:36,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 15:32:36,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 15:32:36,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 15:32:36,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 15:32:36,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 15:32:36,385 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 15:32:36,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 15:32:36,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:32:36,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:32:36,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:36,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:36,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:36,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:36,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:36,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:36,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:36,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:36,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:36,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:36,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:32:36,980 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'sanitizers.log.*' files found 2021-12-16 15:32:36,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:32:36,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:37,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.67: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:32:37,023 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.67: No 'indexer.log*' files found 2021-12-16 15:32:37,023 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.67 2021-12-16 15:32:37,026 | infra | DEBUG | MainThread | [remote_util:connect:207] Connecting to 172.23.110.68 with username: root, password: couchbase 2021-12-16 15:32:37,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-12-16 15:32:37,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /etc/ 2021-12-16 15:32:37,526 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3490] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-12-16 15:32:37,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /cygdrive/c/Windows 2021-12-16 15:32:37,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: uname -m 2021-12-16 15:32:37,569 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/cpuinfo 2021-12-16 15:32:37,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: df -Thl 2021-12-16 15:32:37,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: cat /proc/meminfo 2021-12-16 15:32:37,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname 2021-12-16 15:32:37,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: hostname -d 2021-12-16 15:32:37,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-12-16 15:32:37,703 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:905] 172.23.110.68: No crash files found 2021-12-16 15:32:37,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-12-16 15:32:37,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:37,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:37,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 15:32:37,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000000.txt 2021-12-16 15:32:37,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:37,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:37,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 15:32:38,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-12-16 15:32:38,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 15:32:38,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000002.txt 2021-12-16 15:32:38,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 15:32:38,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000003.txt 2021-12-16 15:32:38,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:32:38,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-12-16 15:32:38,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:38,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000005.txt 2021-12-16 15:32:38,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:38,678 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000006.txt 2021-12-16 15:32:38,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,795 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:38,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000007.txt 2021-12-16 15:32:38,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:38,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:38,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:38,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000008.txt 2021-12-16 15:32:38,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:39,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:39,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:32:39,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-12-16 15:32:39,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:39,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-12-16 15:32:39,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:39,252 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000010.txt 2021-12-16 15:32:39,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-12-16 15:32:39,299 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'sanitizers.log.*' files found 2021-12-16 15:32:39,301 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/babysitter.log 2021-12-16 15:32:39,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/babysitter.log | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' | grep -v 'write traffic will be disabled for this node' 2021-12-16 15:32:39,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3307] Running command on 172.23.110.68: ls /opt/couchbase/var/lib/couchbase/logs/indexer.log* 2021-12-16 15:32:39,361 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:924] 172.23.110.68: No 'indexer.log*' files found 2021-12-16 15:32:39,361 | infra | DEBUG | MainThread | [remote_util:disconnect:220] Disconnecting ssh_client for 172.23.110.68 2021-12-16 15:32:39,362 | test | WARNING | MainThread | [basetestcase:tearDownEverything:615] CLEANUP WAS SKIPPED 2021-12-16 15:32:39,364 | infra | INFO | MainThread | [basetestcase:tearDownEverything:658] ========== tasks in thread pool ========== ======================================================================2021-12-16 15:32:39,364 | infra | INFO | MainThread | [basetestcase:tearDownEverything:660] ========================================== 2021-12-16 15:32:39,364 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-16 15:32:39,364 | infra | INFO | MainThread | [task_manager:shutdown:64] Running TaskManager shutdown 2021-12-16 15:32:39,364 | infra | WARNING | MainThread | [cluster:shutdown:707] Cluster instance shutdown with force FAIL: SystemTestMagma (aGoodDoctor.Hospital.Murphy) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/aGoodDoctor/Hospital.py", line 629, in SystemTestMagma self.assertTrue(rebalance_task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 157146.173s FAILED (failures=1) During the test, Remote Connections: 1373, Disconnections: 1373 SDK Connections: 0, Disconnections: 0 summary so far suite aGoodDoctor.Hospital.Murphy , pass 0 , fail 1 failures so far... aGoodDoctor.Hospital.Murphy.SystemTestMagma testrunner logs, diags and results are available under /data/workspace/magma_temp_job3/logs/testrunner-21-Dec-14_19-53-30/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 43h 39m 31s 2 actionable tasks: 2 executed [description-setter] Could not determine description. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.1.0-1913 [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 3s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) An attempt to send an e-mail to empty list of recipients, ignored. Finished: UNSTABLE